[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 997820643 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003428] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008791] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012069] LSM: Security Framework initializing [ 0.013057] Yama: becoming mindful. [ 0.014040] SELinux: Initializing. [ 0.015068] *** VALIDATE selinux *** [ 0.024291] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029323] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032103] *** VALIDATE tmpfs *** [ 0.034284] *** VALIDATE proc *** [ 0.035262] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.037280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.043865] debug: unmapping init [mem 0xffffffffa3859000-0xffffffffa3860fff] [ 0.046319] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047803] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055269] rcu: Hierarchical SRCU implementation. [ 0.057657] smp: Bringing up secondary CPUs ... [ 0.058585] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.069139] smp: Brought up 1 node, 4 CPUs [ 0.071028] smpboot: Max logical packages: 1 [ 0.072012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.113028] node 0 deferred pages initialised in 39ms [ 0.117525] devtmpfs: initialized [ 0.118223] x86/mm: Memory block size: 128MB [ 0.120000] gcov: version magic: 0x41383552 [ 0.124314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.128159] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.130415] pinctrl core: initialized pinctrl subsystem [ 0.133170] [ 0.133986] ************************************************************* [ 0.137014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140010] ** ** [ 0.143012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146013] ** ** [ 0.149019] ** This means that this kernel is built to expose internal ** [ 0.152011] ** IOMMU data structures, which may compromise security on ** [ 0.155011] ** your system. ** [ 0.157012] ** ** [ 0.160015] ** If you see this message and you are not debugging the ** [ 0.163011] ** kernel, report this immediately to your vendor! ** [ 0.166012] ** ** [ 0.169013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172014] ************************************************************* [ 0.176876] NET: Registered protocol family 16 [ 0.178653] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189637] cpuidle: using governor menu [ 0.190991] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194189] PCI: Using configuration type 1 for base access [ 0.196199] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210343] cryptd: max_cpu_qlen set to 1000 [ 0.212213] ACPI: Added _OSI(Module Device) [ 0.214015] ACPI: Added _OSI(Processor Device) [ 0.216017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225296] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233600] ACPI: Interpreter enabled [ 0.235090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237015] ACPI: Using IOAPIC for interrupt routing [ 0.240122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244545] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257280] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273868] acpiphp: Slot [2] registered [ 0.276129] acpiphp: Slot [5] registered [ 0.278140] acpiphp: Slot [6] registered [ 0.280114] acpiphp: Slot [7] registered [ 0.282116] acpiphp: Slot [8] registered [ 0.283340] acpiphp: Slot [9] registered [ 0.284109] acpiphp: Slot [10] registered [ 0.286114] acpiphp: Slot [3] registered [ 0.288121] acpiphp: Slot [4] registered [ 0.290100] acpiphp: Slot [11] registered [ 0.292100] acpiphp: Slot [12] registered [ 0.294106] acpiphp: Slot [13] registered [ 0.296133] acpiphp: Slot [14] registered [ 0.298147] acpiphp: Slot [15] registered [ 0.300290] acpiphp: Slot [16] registered [ 0.302244] acpiphp: Slot [17] registered [ 0.305214] acpiphp: Slot [18] registered [ 0.307105] acpiphp: Slot [19] registered [ 0.309108] acpiphp: Slot [20] registered [ 0.312731] acpiphp: Slot [21] registered [ 0.315391] acpiphp: Slot [22] registered [ 0.318160] acpiphp: Slot [23] registered [ 0.320115] acpiphp: Slot [24] registered [ 0.323215] acpiphp: Slot [25] registered [ 0.325195] acpiphp: Slot [26] registered [ 0.327194] acpiphp: Slot [27] registered [ 0.329198] acpiphp: Slot [28] registered [ 0.331131] acpiphp: Slot [29] registered [ 0.333330] acpiphp: Slot [30] registered [ 0.335104] acpiphp: Slot [31] registered [ 0.337056] PCI host bridge to bus 0000:00 [ 0.338018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342065] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.348108] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362256] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.366998] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382007] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.391015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.395023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.398021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402366] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406153] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.408189] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.412999] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.419016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.436016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.449119] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.470023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.472000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485726] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.529020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.605014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.628578] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.646013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.655015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.709016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.719797] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.726015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.736138] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.779016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.810516] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.828015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.837015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.883013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.892278] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.895542] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.897327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.900396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.902206] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.908097] iommu: Default domain type: Passthrough [ 0.909514] SCSI subsystem initialized [ 0.911260] ACPI: bus type USB registered [ 0.913130] usbcore: registered new interface driver usbfs [ 0.915076] usbcore: registered new interface driver hub [ 0.916165] usbcore: registered new device driver usb [ 0.919190] pps_core: LinuxPPS API ver. 1 registered [ 0.921010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.924054] PTP clock support registered [ 0.926174] EDAC MC: Ver: 3.0.0 [ 0.927074] PCI: Using ACPI for IRQ routing [ 0.928000] NetLabel: Initializing [ 0.930009] NetLabel: domain hash size = 128 [ 0.932010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.934227] NetLabel: unlabeled traffic allowed by default [ 0.936278] vgaarb: loaded [ 0.938421] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.940015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.946208] clocksource: Switched to clocksource kvm-clock [ 1.091826] VFS: Disk quotas dquot_6.6.0 [ 1.093844] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.097584] *** VALIDATE ramfs *** [ 1.099355] *** VALIDATE hugetlbfs *** [ 1.101456] pnp: PnP ACPI init [ 1.104389] pnp: PnP ACPI: found 6 devices [ 1.122803] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.126500] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.129091] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.131583] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.134426] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.137316] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.140908] NET: Registered protocol family 2 [ 1.144275] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.150253] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.154529] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.160803] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.165507] TCP: Hash tables configured (established 65536 bind 65536) [ 1.169950] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.174220] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.178727] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.183064] NET: Registered protocol family 1 [ 1.186266] RPC: Registered named UNIX socket transport module. [ 1.188754] RPC: Registered udp transport module. [ 1.190443] RPC: Registered tcp transport module. [ 1.192217] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.194501] NET: Registered protocol family 44 [ 1.196450] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.198831] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.201472] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.204477] PCI: CLS 0 bytes, default 64 [ 1.206877] Unpacking initramfs... [ 3.593301] debug: unmapping init [mem 0xffff94c73cc54000-0xffff94c73ffbffff] [ 3.600691] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.603834] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.609330] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.224265] Initialise system trusted keyrings [ 4.226268] Key type blacklist registered [ 4.228592] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.239408] zbud: loaded [ 4.243074] *** VALIDATE nfs *** [ 4.244524] *** VALIDATE nfs4 *** [ 4.246816] pstore: using deflate compression [ 4.251311] Platform Keyring initialized [ 4.387835] NET: Registered protocol family 38 [ 4.389919] Key type asymmetric registered [ 4.393034] Asymmetric key parser 'x509' registered [ 4.395457] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.401192] io scheduler mq-deadline registered [ 4.403475] io scheduler kyber registered [ 4.406106] io scheduler bfq registered [ 4.408433] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.412822] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.417467] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.423360] ACPI: Power Button [PWRF] [ 4.430564] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.439902] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.553955] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.570873] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.628737] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.660606] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.985805] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.994301] Non-volatile memory driver v1.3 [ 4.995907] Linux agpgart interface v0.103 [ 5.028720] virtio_blk virtio1: [vda] 134016 512-byte logical blocks (68.6 MB/65.4 MiB) [ 5.031289] vda: detected capacity change from 0 to 68616192 [ 5.328894] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.433288] vdb: detected capacity change from 0 to 1073741824 [ 5.450410] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.453901] vdc: detected capacity change from 0 to 2621440000 [ 5.469453] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.472027] vdd: detected capacity change from 0 to 2621440000 [ 5.484054] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.489254] vde: detected capacity change from 0 to 4294967296 [ 5.507955] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.510658] vdf: detected capacity change from 0 to 4294967296 [ 5.517188] libphy: Fixed MDIO Bus: probed [ 5.521920] usbcore: registered new interface driver usbserial_generic [ 5.524183] usbserial: USB Serial support registered for generic [ 5.526319] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.530973] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.535063] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.541461] mousedev: PS/2 mouse device common for all mice [ 5.550196] rtc_cmos 00:05: RTC can wake from S4 [ 5.558177] rtc_cmos 00:05: registered as rtc0 [ 5.561887] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.568097] intel_pstate: CPU model not supported [ 5.573544] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.577828] hid: raw HID events driver (C) Jiri Kosina [ 5.590893] usbcore: registered new interface driver usbhid [ 5.598055] usbhid: USB HID core driver [ 5.604135] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.608427] drop_monitor: Initializing network drop monitor service [ 5.614585] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.617844] Initializing XFRM netlink socket [ 5.628156] NET: Registered protocol family 10 [ 5.635861] Segment Routing with IPv6 [ 5.638251] NET: Registered protocol family 17 [ 5.641229] mpls_gso: MPLS GSO support [ 5.647452] RAS: Correctable Errors collector initialized. [ 5.650223] AVX version of gcm_enc/dec engaged. [ 5.652088] AES CTR mode by8 optimization enabled [ 5.765387] sched_clock: Marking stable (5765368788, 0)->(7040752850, -1275384062) [ 5.772599] registered taskstats version 1 [ 5.774868] Loading compiled-in X.509 certificates [ 5.777410] zswap: loaded using pool lzo/zbud [ 5.802506] Key type big_key registered [ 5.815668] Key type encrypted registered [ 5.817682] ima: No TPM chip found, activating TPM-bypass! [ 5.820380] ima: Allocated hash algorithm: sha1 [ 5.822464] ima: No architecture policies found [ 5.824600] evm: Initialising EVM extended attributes: [ 5.831776] evm: security.selinux [ 5.836305] evm: security.ima [ 5.840357] evm: security.capability [ 5.845420] evm: HMAC attrs: 0x1 [ 5.859270] rtc_cmos 00:05: setting system clock to 2026-04-30 18:48:52 UTC (1777574932) [ 5.870859] debug: unmapping init [mem 0xffffffffa4803000-0xffffffffa49fffff] [ 5.878884] debug: unmapping init [mem 0xffffffffa3582000-0xffffffffa3858fff] [ 5.892165] Write protecting the kernel read-only data: 28672k [ 5.898486] debug: unmapping init [mem 0xffffffffa1c03000-0xffffffffa1dfffff] [ 5.903256] debug: unmapping init [mem 0xffffffffa2514000-0xffffffffa25fffff] [ 5.952235] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.974195] systemd[1]: Detected virtualization kvm. [ 5.978922] systemd[1]: Detected architecture x86-64. [ 5.984169] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.031105] systemd[1]: No hostname configured. [ 6.036136] systemd[1]: Set hostname to . [ 6.041425] random: systemd: uninitialized urandom read (16 bytes read) [ 6.047649] systemd[1]: Initializing machine ID from random generator. [ 6.216923] random: ln: uninitialized urandom read (6 bytes read) [ 6.349318] random: systemd: uninitialized urandom read (16 bytes read) [ 6.353880] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.362576] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.369218] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started Journal Service. [ 7.448477] device-mapper: uevent: version 1.0.3 [ 7.453609] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.662096] virtio_net virtio0 ens2: renamed from eth0 [ 9.720927] random: fast init done [ 11.225203] scsi host0: ata_piix [ 11.248377] scsi host1: ata_piix [ 11.255138] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.266107] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.577712] random: crng init done [ 15.582369] random: 7 urandom warning(s) missed due to ratelimiting [ 18.264032] dracut-initqueue[582]: RTNETLINK answers: File exists [ 19.091060] hrtimer: interrupt took 5066048 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.334781] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.489118] printk: systemd: 28 output lines suppressed due to ratelimiting [ 24.118787] SELinux: Disabled at runtime. [ 24.190811] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.201091] systemd[1]: Detected virtualization kvm. [ 24.203229] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.297869] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.301974] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.311189] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.316767] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.324762] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.333387] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.347773] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 25.395435] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.933052] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.360323] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.372636] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 33.680080] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.637729] EDAC sbridge: Ver: 1.1.2 [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 37.924139] Key type dns_resolver registered [[ 38.307250] NFS: Registering the id_resolver key type [ 38.312120] Key type id_resolver registered [ 38.313957] Key type id_legacy registered *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 115.644393] libcfs: loading out-of-tree module taints kernel. [ 115.703434] Key type ._llcrypt registered [ 115.707478] Key type .llcrypt registered [ 115.890995] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 137.875948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 139.530870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 139.539808] alg: No test for adler32 (adler32-zlib) [ 141.397772] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 142.674088] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 144.351453] Key type lgssc registered [ 146.538672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 163.988246] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 212.683417] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 228.685993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 228.719562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.116702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 230.150095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 230.305342] Lustre: lustre-MDT0000: new disk, initializing [ 230.404793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.436405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 236.106852] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.376757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.500237] Lustre: 6512:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 251.556894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 251.565959] Lustre: Skipped 1 previous similar message [ 251.745950] Lustre: lustre-MDT0001: new disk, initializing [ 251.824944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.850941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 251.861773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 256.646943] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.591618] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 272.334608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.683931] Lustre: lustre-OST0000: new disk, initializing [ 272.691311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 272.791985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 278.096351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 278.101252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 278.287817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 280.083325] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.416444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.554728] Lustre: lustre-OST0001: new disk, initializing [ 296.560541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 296.612945] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 304.460197] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 305.731525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 305.750144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 305.849243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 319.403954] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 326.223914] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 339.012299] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 344.520837] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 350.085951] Lustre: DEBUG MARKER: Client: 2.17.52.53 [ 353.758400] Lustre: DEBUG MARKER: MDS: 2.17.52.53 [ 357.218780] Lustre: DEBUG MARKER: OSS: 2.17.52.53 [ 359.330913] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Apr 30 14:54:43 EDT 2026 [ 380.719119] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 383.012874] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 403.425094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.427591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 403.446915] Lustre: Skipped 1 previous similar message [ 403.481997] Lustre: Skipped 3 previous similar messages [ 407.139060] Lustre: server umount lustre-MDT0000 complete [ 411.007922] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575337 with bad export cookie 18099712492749780144 [ 411.029040] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 411.034763] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 413.664293] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 413.664485] LustreError: 6517:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.686538] Lustre: Skipped 2 previous similar messages [ 413.706070] LustreError: 6517:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 413.727453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 413.739646] Lustre: Skipped 1 previous similar message [ 416.671915] LustreError: 10336:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.672969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 416.697937] Lustre: Skipped 1 previous similar message [ 417.592932] Lustre: server umount lustre-MDT0001 complete [ 423.386594] Lustre: server umount lustre-OST0000 complete [ 427.922584] Lustre: server umount lustre-OST0001 complete [ 436.461459] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 446.616796] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 490.858823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.281455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 491.360146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 491.449393] Lustre: lustre-MDT0000: new disk, initializing [ 491.585561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.607646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 496.691689] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 509.521474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.645748] Lustre: 14990:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 509.704770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 509.708947] Lustre: Skipped 1 previous similar message [ 509.780402] Lustre: lustre-MDT0001: new disk, initializing [ 509.869644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 509.952114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 509.973342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 514.832088] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 523.671086] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 528.164767] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 535.390134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 535.651433] Lustre: lustre-OST0000: new disk, initializing [ 535.658108] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 535.758318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.178790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 537.189415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 537.338528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 542.857164] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 551.067374] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 556.907663] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 557.279799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 564.564361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 564.971208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 568.295204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 568.302347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.323282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 568.347879] Lustre: Skipped 2 previous similar messages [ 568.376425] Lustre: Skipped 1 previous similar message [ 573.420154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.430388] Lustre: Skipped 1 previous similar message [ 574.148551] Lustre: server umount lustre-OST0000 complete [ 578.528035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.551326] Lustre: Skipped 1 previous similar message [ 583.649819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.657237] Lustre: Skipped 3 previous similar messages [ 592.351340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 592.612103] Lustre: server umount lustre-MDT0000 complete [ 593.894632] LustreError: 15002:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.912706] LustreError: 15002:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 596.733506] LustreError: 14981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575523 with bad export cookie 18099712492749781677 [ 596.736995] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.742606] LustreError: 14981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 597.179648] Lustre: server umount lustre-MDT0001 complete [ 603.403342] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:58:48 (1777575528) [ 622.766979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.500297] LustreError: 19462:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.573667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.317933] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 628.704450] LustreError: 19463:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.161310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.475166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 648.587698] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 654.593242] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 658.250332] Lustre: Failing over lustre-MDT0000 [ 658.958428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.998352] LustreError: 19458:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.039535] LustreError: 19458:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 659.405671] Lustre: server umount lustre-MDT0000 complete [ 678.367196] Lustre: 3662:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777575589/real 1777575589] req@ffff94c6818e1500 x1863922555719168/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777575605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 678.431369] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.460649] LustreError: 20454:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.477211] LustreError: 20454:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 681.080622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.110781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.119177] Lustre: Skipped 1 previous similar message [ 689.194752] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 694.246805] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 694.254640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 694.288831] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 694.761737] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 703.900434] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 709.602071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 709.615144] LustreError: Skipped 1 previous similar message [ 709.636918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.664176] Lustre: Skipped 1 previous similar message [ 709.675084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.678191] Lustre: Skipped 2 previous similar messages [ 715.130253] Lustre: server umount lustre-MDT0000 complete [ 717.804740] LustreError: 19463:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.830599] LustreError: 19463:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 719.559336] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575646 with bad export cookie 18099712492749783091 [ 719.571245] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.575361] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 720.056777] Lustre: server umount lustre-MDT0001 complete [ 730.142035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.425099] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 747.933896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.691375] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 761.149712] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 766.275378] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 769.038715] Lustre: Failing over lustre-MDT0000 [ 769.339154] Lustre: server umount lustre-MDT0000 complete [ 774.113032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.113762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.132970] Lustre: Skipped 1 previous similar message [ 785.376759] LustreError: 22466:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 785.405753] LustreError: 22466:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 11 previous similar messages [ 789.535105] Lustre: 3662:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777575700/real 1777575700] req@ffff94c7bb535180 x1863922555749760/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777575716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 789.572178] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.768261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.217717] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 805.350497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 805.368286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 805.382060] Lustre: Skipped 1 previous similar message [ 805.403926] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 805.570897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 814.334270] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 819.680673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.685411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 819.702397] Lustre: Skipped 2 previous similar messages [ 819.709651] Lustre: Skipped 2 previous similar messages [ 824.708845] Lustre: server umount lustre-MDT0000 complete [ 829.006300] LustreError: 23823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575755 with bad export cookie 18099712492749784029 [ 829.016562] LustreError: 23823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 829.019081] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 829.384813] Lustre: server umount lustre-MDT0001 complete [ 839.090980] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:02:43 (1777575763) [ 857.844138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 870.758444] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 922.031845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 934.413517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.694831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 934.739460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 934.861649] Lustre: lustre-MDT0000: new disk, initializing [ 934.996367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.007782] Lustre: Skipped 3 previous similar messages [ 935.050264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 940.024431] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 952.559412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.718515] Lustre: 28467:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 953.133068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 958.583677] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 966.999690] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 971.463408] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 979.195293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.509713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 981.543155] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 981.644395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 981.673326] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 986.618442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 994.810476] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1001.623876] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1027.557392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1027.563073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.577934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1027.582716] Lustre: Skipped 2 previous similar messages [ 1033.598852] Lustre: server umount lustre-OST0000 complete [ 1044.574687] Lustre: server umount lustre-MDT0000 complete [ 1048.552963] LustreError: 28474:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.573946] LustreError: 28474:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 1049.017478] LustreError: 28457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777575975 with bad export cookie 18099712492749784435 [ 1049.041532] LustreError: 28457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1049.049952] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.709894] Lustre: server umount lustre-MDT0001 complete [ 1062.202420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1065.500546] Key type lgssc unregistered [ 1065.878446] LNet: 32059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.893545] LNetError: 32059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.916368] LNet: Removed LNI 192.168.203.155@tcp [ 1067.202922] Key type .llcrypt unregistered [ 1067.206395] Key type ._llcrypt unregistered [ 1095.155670] Key type ._llcrypt registered [ 1095.158778] Key type .llcrypt registered [ 1095.261601] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1096.437279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.491680] alg: No test for adler32 (adler32-zlib) [ 1097.762825] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 1098.075050] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1099.767213] Key type lgssc registered [ 1101.351856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.738778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.767983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.395811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.176913] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1120.741421] LustreError: 33456:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.759393] LustreError: 33456:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1125.506754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.879735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1130.862695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1137.068299] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.150148] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1150.134646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.438227] LustreError: 35052:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.640440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.774521] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1155.835610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1158.263419] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1164.984824] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1181.278911] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:08:25 (1777576105) [ 1183.459743] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1185.695709] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1194.855829] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:08:39 (1777576119) [ 1206.233206] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:08:50 (1777576130) [ 1207.747394] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1218.650177] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1218.660475] Lustre: Skipped 2 previous similar messages [ 1220.926515] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:09:05 (1777576145) [ 1223.053439] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1223.065447] Lustre: Skipped 2 previous similar messages [ 1232.029620] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1232.035875] Lustre: Skipped 4 previous similar messages [ 1248.178608] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:09:32 (1777576172) [ 1251.523837] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1255.275340] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1266.075842] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:09:50 (1777576190) [ 1268.301525] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1268.310511] Lustre: Skipped 6 previous similar messages [ 1284.895565] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:10:09 (1777576209) [ 1304.555576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.565099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1304.580500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1304.588474] Lustre: Skipped 1 previous similar message [ 1306.806724] Lustre: server umount lustre-OST0000 complete [ 1314.785243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.790761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.817683] Lustre: Skipped 2 previous similar messages [ 1314.855427] Lustre: Skipped 1 previous similar message [ 1317.386637] Lustre: server umount lustre-MDT0000 complete [ 1319.904759] LustreError: 33456:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.920778] LustreError: 33456:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1321.268136] LustreError: 36680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576247 with bad export cookie 7728920400522199742 [ 1321.269502] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.280877] LustreError: 36680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1321.725526] Lustre: server umount lustre-MDT0001 complete [ 1332.550093] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1336.202716] Key type lgssc unregistered [ 1336.462522] LNet: 40961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.470982] LNetError: 40961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.511238] LNet: Removed LNI 192.168.203.155@tcp [ 1338.416583] Key type .llcrypt unregistered [ 1338.417934] Key type ._llcrypt unregistered [ 1359.256345] Key type ._llcrypt registered [ 1359.264486] Key type .llcrypt registered [ 1359.414953] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1377.140571] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1378.550845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1378.633093] alg: No test for adler32 (adler32-zlib) [ 1379.720663] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 1379.939816] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1381.663459] Key type lgssc registered [ 1383.036370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.677923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1422.716544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.110585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1424.137278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1424.311233] Lustre: lustre-MDT0000: new disk, initializing [ 1424.500674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.537337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1429.532914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1440.260155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.329171] Lustre: 44085:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1440.432032] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1440.442296] Lustre: Skipped 1 previous similar message [ 1440.569950] Lustre: lustre-MDT0001: new disk, initializing [ 1440.676509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1440.735965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1440.746845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1445.918669] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1454.454057] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1459.162875] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1466.996569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.414431] Lustre: lustre-OST0000: new disk, initializing [ 1467.419426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1467.501457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.982164] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1477.730899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1477.743388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1477.846156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1482.198958] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1487.707136] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1487.967897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1492.514590] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1492.780555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1498.599442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1498.603594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.610877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1498.624078] Lustre: Skipped 1 previous similar message [ 1498.635351] Lustre: Skipped 1 previous similar message [ 1503.731639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1503.743514] Lustre: Skipped 1 previous similar message [ 1508.833323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.343153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1509.589325] Lustre: server umount lustre-OST0000 complete [ 1513.954492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.978657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.986584] Lustre: Skipped 2 previous similar messages [ 1519.075247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.081758] Lustre: Skipped 1 previous similar message [ 1520.165653] Lustre: server umount lustre-MDT0000 complete [ 1524.254604] LustreError: 44078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576450 with bad export cookie 14856614831672755442 [ 1524.264969] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1524.270372] LustreError: 44078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1524.714265] Lustre: server umount lustre-MDT0001 complete [ 1536.489399] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:14:20 (1777576460) [ 1544.557983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.228541] LustreError: 48141:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.252399] LustreError: 48141:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1545.329476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.516858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1550.816351] LustreError: 48142:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.930488] LustreError: 48141:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.678831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.037086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1560.206219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.076571] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1571.701914] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1576.537628] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1584.410845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1584.882793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.086717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1597.319387] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1602.648081] LustreError: 50332:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1603.867876] Lustre: Modifying parameter general.timeout=1 in log params [ 1605.636403] Lustre: Modifying parameter general.timeout=3 in log params [ 1605.639625] Lustre: Skipped 1 previous similar message [ 1608.443991] Lustre: Modifying parameter general.timeout=6 in log params [ 1608.449602] Lustre: Skipped 2 previous similar messages [ 1613.244431] Lustre: Modifying parameter general.timeout=11 in log params [ 1613.251380] Lustre: Skipped 4 previous similar messages [ 1621.872443] Lustre: Modifying parameter general.timeout=20 in log params [ 1621.875024] Lustre: Skipped 8 previous similar messages [ 1638.623561] Lustre: Modifying parameter general.timeout=37 in log params [ 1638.634198] Lustre: Skipped 16 previous similar messages [ 1671.221596] Lustre: Modifying parameter general.timeout=71 in log params [ 1671.229194] Lustre: Skipped 33 previous similar messages [ 1736.450877] Lustre: Disabling parameter general.timeout= in log params [ 1736.464540] Lustre: Skipped 58 previous similar messages [ 1740.504684] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:17:44 (1777576664) [ 1743.251130] Lustre: 57285:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1744.995233] Lustre: 57383:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1748.207515] Lustre: 57579:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1748.226892] Lustre: 57579:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1752.344049] Lustre: 57787:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1752.361058] Lustre: 57787:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1781.989059] Lustre: 59147:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1782.017863] LustreError: 59147:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1786.962827] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:18:31 (1777576711) [ 1800.682804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.689708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.698288] Lustre: Skipped 2 previous similar messages [ 1800.698985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1800.723563] LustreError: Skipped 1 previous similar message [ 1807.061852] Lustre: server umount lustre-MDT0000 complete [ 1810.914303] LustreError: 49252:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.949332] LustreError: 49252:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1810.973252] LustreError: 50826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576737 with bad export cookie 14856614831672756324 [ 1810.979813] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.993984] LustreError: 50826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1811.372178] Lustre: server umount lustre-MDT0001 complete [ 1826.015397] Lustre: server umount lustre-OST0000 complete [ 1853.752423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.114811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1854.437287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.592420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1870.107298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.210246] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1870.243752] Lustre: 61677:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1870.360958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1870.363577] Lustre: Skipped 5 previous similar messages [ 1875.715863] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1882.328849] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.405351] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1894.287900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.541927] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1894.890377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.897683] Lustre: Skipped 1 previous similar message [ 1901.386266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1906.963320] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1908.799522] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1908.806638] Lustre: Skipped 23 previous similar messages [ 1908.864432] Lustre: 63906:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1917.408900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1917.424459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.440610] Lustre: Skipped 1 previous similar message [ 1923.500665] Lustre: server umount lustre-OST0000 complete [ 1928.160679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.190939] Lustre: Skipped 1 previous similar message [ 1934.383346] Lustre: server umount lustre-MDT0000 complete [ 1938.402803] LustreError: 61690:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.442399] LustreError: 61690:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1938.589912] LustreError: 61671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777576865 with bad export cookie 14856614831672759390 [ 1938.614504] LustreError: 61671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1938.616344] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1938.999445] Lustre: server umount lustre-MDT0001 complete [ 1952.709165] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1955.848858] Key type lgssc unregistered [ 1956.247654] LNet: 65098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.261601] LNetError: 65098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.278749] LNet: Removed LNI 192.168.203.155@tcp [ 1957.145717] Key type .llcrypt unregistered [ 1957.147494] Key type ._llcrypt unregistered [ 1982.283277] Key type ._llcrypt registered [ 1982.285171] Key type .llcrypt registered [ 1982.454840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1983.637198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1983.702244] alg: No test for adler32 (adler32-zlib) [ 1984.829515] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 1985.141718] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1986.911904] Key type lgssc registered [ 1988.472267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1996.825492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1996.841269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.544514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.106268] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2008.040795] LustreError: 66499:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.066607] LustreError: 66499:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2013.153758] LustreError: 66498:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.552658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.318814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2018.613957] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2023.435968] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2027.254649] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2035.832612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2036.345884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.918706] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2050.135823] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2058.824877] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2068.455134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.457076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.479567] Lustre: Skipped 2 previous similar messages [ 2068.497823] Lustre: Skipped 2 previous similar messages [ 2073.570484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2073.586270] Lustre: Skipped 1 previous similar message [ 2074.340510] Lustre: server umount lustre-MDT0000 complete [ 2078.151412] LustreError: 66479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577004 with bad export cookie 621293710241321276 [ 2078.153755] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2078.171723] LustreError: 66479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2078.509805] Lustre: server umount lustre-MDT0001 complete [ 2093.123779] Lustre: server umount lustre-OST0000 complete [ 2122.168374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.407436] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2122.436284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2122.649449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.985340] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2135.614575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.727182] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2135.766840] Lustre: 71031:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2136.216404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.862906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2146.322850] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2150.603197] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2156.706174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2156.852382] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2157.271486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.296688] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2168.724891] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2170.666578] Lustre: 73265:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2172.810370] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2182.112097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.134963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.162601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.627142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2184.813100] Lustre: server umount lustre-OST0000 complete [ 2192.353125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.356553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.404728] Lustre: Skipped 1 previous similar message [ 2192.408296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.412506] Lustre: Skipped 1 previous similar message [ 2195.196380] Lustre: server umount lustre-MDT0000 complete [ 2197.984643] LustreError: 71044:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.012833] LustreError: 71044:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2199.628115] LustreError: 72671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577126 with bad export cookie 621293710241322116 [ 2199.630901] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.639702] LustreError: 72671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2200.144117] Lustre: server umount lustre-MDT0001 complete [ 2212.061302] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2214.945635] Key type lgssc unregistered [ 2215.309838] LNet: 74460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.321344] LNetError: 74460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.349613] LNet: Removed LNI 192.168.203.155@tcp [ 2215.902876] Key type .llcrypt unregistered [ 2215.905186] Key type ._llcrypt unregistered [ 2240.578082] Key type ._llcrypt registered [ 2240.580281] Key type .llcrypt registered [ 2240.683527] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2241.800218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.941112] alg: No test for adler32 (adler32-zlib) [ 2243.131375] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 2243.430953] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2245.111173] Key type lgssc registered [ 2246.758328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2256.849594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2256.889213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.290404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.522841] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2267.635714] LustreError: 75860:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.666090] LustreError: 75860:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2272.764687] LustreError: 75861:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.599430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.975913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2280.995594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2287.999994] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2292.016194] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2299.168026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.684764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2305.372809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2311.048679] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2317.256368] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2318.216511] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2319.410729] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2322.224934] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2322.244691] Lustre: Skipped 2 previous similar messages [ 2327.042883] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2327.049734] Lustre: Skipped 4 previous similar messages [ 2335.257435] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2335.265250] Lustre: Skipped 8 previous similar messages [ 2351.280560] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2351.290704] Lustre: Skipped 16 previous similar messages [ 2383.858310] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2383.863964] Lustre: Skipped 33 previous similar messages [ 2422.328931] Lustre: 83622:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2425.449354] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:29:09 (1777577349) [ 2427.487050] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2429.908765] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:29:14 (1777577354) [ 2435.041147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2435.045979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.079647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2437.091294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2437.100911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2437.108381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.945712] Lustre: server umount lustre-OST0000 complete [ 2445.792260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.802425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.820370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.831643] Lustre: Skipped 1 previous similar message [ 2451.953316] Lustre: server umount lustre-MDT0000 complete [ 2452.448488] LustreError: 75857:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.475526] LustreError: 75857:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2456.649947] LustreError: 75840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577383 with bad export cookie 18118359604752359569 [ 2456.651826] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.676199] LustreError: 75840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2457.207954] Lustre: server umount lustre-MDT0001 complete [ 2470.252097] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2473.665364] Key type lgssc unregistered [ 2474.072604] LNet: 84855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.084739] LNetError: 84855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.117884] LNet: Removed LNI 192.168.203.155@tcp [ 2475.003963] Key type .llcrypt unregistered [ 2475.008710] Key type ._llcrypt unregistered [ 2479.978802] Key type ._llcrypt registered [ 2479.981351] Key type .llcrypt registered [ 2480.100551] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2485.265304] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules [ 2486.544680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2486.840101] alg: No test for adler32 (adler32-zlib) [ 2487.967825] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 2488.268770] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2490.055240] Key type lgssc registered [ 2491.343518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2492.641456] LustreError: 86026:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 2514.582668] LustreError: 86026:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2514.631640] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2525.338509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2525.362621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.260629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.642191] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2536.424358] LustreError: 87031:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.445107] LustreError: 87031:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2541.551101] LustreError: 87032:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.243644] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:31:08 (1777577468) [ 2546.661663] LustreError: 87031:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.092208] LustreError: 87063:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2550.114799] LustreError: 87063:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2550.121696] LustreError: 87063:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2550.451625] Lustre: server umount lustre-MDT0000 complete [ 2566.684683] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2570.134819] Key type lgssc unregistered [ 2570.483691] LNet: 88524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2570.494874] LNetError: 88524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2570.505533] LNet: Removed LNI 192.168.203.155@tcp [ 2571.403252] Key type .llcrypt unregistered [ 2571.418217] Key type ._llcrypt unregistered [ 2590.425414] Key type ._llcrypt registered [ 2590.428268] Key type .llcrypt registered [ 2590.542514] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2591.497630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2591.528243] alg: No test for adler32 (adler32-zlib) [ 2592.874039] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 2593.285804] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2595.007127] Key type lgssc registered [ 2596.215806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2605.500912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2605.527165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.183623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.714546] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2617.343747] LustreError: 89727:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.371754] LustreError: 89727:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2620.993723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.607862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2625.760634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2630.876280] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2635.196977] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2641.269233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2641.848051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.005258] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2654.189264] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2710.077250] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:33:54 (1777577634) [ 2712.756185] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2715.370795] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:33:59 (1777577639) [ 2723.808356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2723.818848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.849879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2723.870550] Lustre: Skipped 2 previous similar messages [ 2726.198055] Lustre: server umount lustre-MDT0000 complete [ 2728.930070] LustreError: 92249:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.954841] LustreError: 92249:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2729.994974] LustreError: 89709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577656 with bad export cookie 16693237206771070687 [ 2730.002395] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.545333] Lustre: server umount lustre-MDT0001 complete [ 2745.188898] Lustre: server umount lustre-OST0000 complete [ 2757.289212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.779419] LustreError: 93681:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.025357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.828958] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2763.235608] LustreError: 93682:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.264705] LustreError: 93682:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2768.360115] LustreError: 93681:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.456219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.975161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.533385] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2781.645527] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2785.437329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2798.215811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.464450] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2798.471581] LustreError: 93671:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2798.509745] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2798.517701] LustreError: 95538:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -98 [ 2798.525694] LustreError: 95538:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 2798.538202] LustreError: 95538:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2798.622606] Lustre: server umount lustre-OST0000 complete [ 2798.625114] LustreError: 95538:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2807.093461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2807.257355] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2807.266703] LustreError: 93671:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2807.286830] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2807.295425] LustreError: 96004:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -98 [ 2807.304600] LustreError: 96004:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 2807.310884] LustreError: 96004:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2807.378890] Lustre: server umount lustre-OST0000 complete [ 2807.384072] LustreError: 96004:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2815.902769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.973717] LustreError: 96522:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.061354] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2816.067127] Lustre: Found index 0 for lustre-OST0000, updating log [ 2816.072400] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2816.138592] Lustre: lustre-OST0000: new disk, initializing [ 2816.143475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2816.383876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.933790] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2825.599906] Lustre: Failing over lustre-OST0000 [ 2825.753557] Lustre: server umount lustre-OST0000 complete [ 2832.353456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2832.365354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.380690] Lustre: Skipped 2 previous similar messages [ 2832.386729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.888498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.890920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.573737] Lustre: server umount lustre-MDT0000 complete [ 2839.020855] LustreError: 93676:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.039637] LustreError: 93676:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2839.305045] LustreError: 93662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777577765 with bad export cookie 16693237206771073431 [ 2839.316029] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2839.328262] LustreError: 93662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2848.875526] Lustre: server umount lustre-MDT0001 complete [ 2858.150814] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:36:22 (1777577782) [ 2876.309734] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2887.844252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.455405] LustreError: 98853:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.659973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.639619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2905.258406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.050374] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2914.460343] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2922.328949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2923.007598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.015126] Lustre: Skipped 1 previous similar message [ 2923.027329] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2930.338835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2931.176384] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2931.211807] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2931.215692] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2940.191956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2940.322042] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2940.422837] Lustre: lustre-OST0001: new disk, initializing [ 2940.434261] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2946.722996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2947.167383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2947.176664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2947.258793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2954.762142] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2960.234181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2970.207179] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 2970.210461] Lustre: Skipped 1 previous similar message [ 2972.639896] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2972.653386] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.294088] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 2975.301273] Lustre: Skipped 4 previous similar messages [ 2976.174832] Lustre: server umount lustre-MDT0001 complete [ 2978.282408] LustreError: 98848:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.308205] LustreError: 98848:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 2986.267377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.427320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2986.436566] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2986.495829] Lustre: 98841:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2987.416989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2987.425774] Lustre: Skipped 1 previous similar message [ 2988.098957] LustreError: 102647:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2988.109993] LustreError: 102647:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2988.131083] LustreError: 98845:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2988.272201] LustreError: 102658:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -17 [ 2988.287783] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2988.287783] [ 2992.621066] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2992.961372] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3004.037106] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 15:38:48 (1777577928) [ 3024.408107] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3043.587798] Lustre: 104689:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3071.476923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3076.003436] Lustre: 105825:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3186.657955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3186.667842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.678160] Lustre: Skipped 2 previous similar messages [ 3186.686451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3192.212647] Lustre: server umount lustre-MDT0000 complete [ 3192.297617] LustreError: 98848:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.325347] LustreError: 98848:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 3196.651690] LustreError: 101824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777578123 with bad export cookie 16693237206771074432 [ 3196.654456] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.668499] LustreError: 101824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3196.934958] Lustre: server umount lustre-MDT0001 complete [ 3211.850501] Lustre: server umount lustre-OST0000 complete [ 3213.859329] Lustre: 89038:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777578124/real 1777578124] req@ffff94c68b3a4e00 x1863925124940288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777578140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3213.893338] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.910615] Lustre: Skipped 3 previous similar messages [ 3215.903824] Lustre: server umount lustre-OST0001 complete [ 3222.328447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3233.121981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3241.765568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3251.814039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3274.274380] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3285.953422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.990767] Lustre: lustre-MDT0000: reset Object Index mappings [ 3286.628310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3290.923858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3300.786907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.801655] Lustre: lustre-MDT0001: reset Object Index mappings [ 3306.730361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3309.840651] Lustre: 110885:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3317.113488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.571111] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3326.377735] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3332.123201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3337.499546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3343.876525] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3343.972201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3345.325634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3354.329159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3358.666435] Lustre: 112729:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3429.023179] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:45:53 (1777578353) [ 3454.436359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.440762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.471413] Lustre: Skipped 3 previous similar messages [ 3454.486588] Lustre: Skipped 7 previous similar messages [ 3460.615836] Lustre: server umount lustre-MDT0000 complete [ 3464.625307] LustreError: 109753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777578391 with bad export cookie 16693237206771121318 [ 3464.634040] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3464.657600] LustreError: 109753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3464.683181] LustreError: 109768:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3464.727463] LustreError: 109768:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 10 previous similar messages [ 3471.183037] Lustre: server umount lustre-MDT0001 complete [ 3481.858628] Lustre: server umount lustre-OST0000 complete [ 3493.292949] Lustre: server umount lustre-OST0001 complete [ 3500.373588] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3511.063941] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3555.562741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.971949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3556.029250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3556.194785] Lustre: lustre-MDT0000: new disk, initializing [ 3556.286705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.296919] Lustre: Skipped 3 previous similar messages [ 3556.317052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.668248] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3567.940569] Lustre: server umount lustre-MDT0000 complete [ 3577.390124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.652953] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3577.671243] Lustre: 116951:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3577.694377] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3582.769399] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3589.285424] Lustre: server umount lustre-MDT0000 complete [ 3601.525868] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:48:46 (1777578526) [ 3603.929759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3606.398814] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:48:50 (1777578530) [ 3627.763973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3637.442308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.076202] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3651.331105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.459952] Lustre: 118968:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3651.554734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3651.562284] Lustre: Skipped 1 previous similar message [ 3651.654161] Lustre: lustre-MDT0001: new disk, initializing [ 3651.818406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3651.829357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3655.797183] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3660.297337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3667.850124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3668.204316] Lustre: lustre-OST0000: new disk, initializing [ 3668.212982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3669.642949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3669.654901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3669.834140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3677.032386] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3689.695528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3689.885736] Lustre: lustre-OST0001: new disk, initializing [ 3689.890957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3691.435416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3691.450133] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3691.570057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3696.410400] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3707.112121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3711.469851] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4155.471410] Lustre: 127777:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4155.493334] Lustre: 127777:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4155.508938] Lustre: 127777:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4155.515969] Lustre: 127777:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4155.526485] Lustre: 127777:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4155.537624] Lustre: 127777:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4500.317860] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:03:44 (1777579424) [ 4517.351218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.351675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4517.378493] Lustre: Skipped 2 previous similar messages [ 4517.383279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.383289] Lustre: Skipped 6 previous similar messages [ 4521.861768] Lustre: server umount lustre-MDT0000 complete [ 4522.470505] LustreError: 122197:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.494074] LustreError: 122197:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4525.723801] LustreError: 118962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777579452 with bad export cookie 16693237206771191878 [ 4525.728313] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.744032] LustreError: 118962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4527.593340] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.606635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4527.607557] Lustre: Skipped 4 previous similar messages [ 4527.615677] Lustre: Skipped 4 previous similar messages [ 4535.798695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4535.814125] Lustre: Skipped 3 previous similar messages [ 4540.383312] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4540.721540] Lustre: server umount lustre-MDT0001 complete [ 4552.855567] Lustre: server umount lustre-OST0000 complete [ 4564.282060] Lustre: server umount lustre-OST0001 complete [ 4571.289809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4587.675469] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4634.314703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4634.677370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4634.739406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4634.860874] Lustre: lustre-MDT0000: new disk, initializing [ 4635.095909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.103327] Lustre: Skipped 5 previous similar messages [ 4635.173993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4640.064406] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4652.794347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.961559] Lustre: 130841:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.078255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4653.090739] Lustre: Skipped 1 previous similar message [ 4653.299768] Lustre: lustre-MDT0001: new disk, initializing [ 4653.507410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4653.530709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4658.102086] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4666.940912] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4671.691662] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4679.810856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4680.231125] Lustre: lustre-OST0000: new disk, initializing [ 4680.240786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4682.145245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4682.161435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4682.305156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4686.181383] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4695.752492] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4708.463466] Lustre: lustre-MDD0000: changelog on [ 4711.937589] Lustre: lustre-MDD0001: changelog on [ 6984.470504] Lustre: lustre-MDD0001: changelog off [ 6986.151311] Lustre: 130849:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6987.013495] Lustre: lustre-MDD0000: changelog off [ 6996.454317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6996.467444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6996.483130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6996.489851] LustreError: Skipped 1 previous similar message [ 6998.913390] Lustre: server umount lustre-OST0000 complete [ 7003.617752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7003.622709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.629963] Lustre: Skipped 1 previous similar message [ 7003.632432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.635069] Lustre: Skipped 1 previous similar message [ 7006.688519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.350851] Lustre: server umount lustre-MDT0000 complete [ 7011.808927] LustreError: 133954:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7011.837284] LustreError: 133954:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 7013.478461] LustreError: 132580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777581940 with bad export cookie 16693237206772015911 [ 7013.483464] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7013.486294] LustreError: 132580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7013.839336] Lustre: server umount lustre-MDT0001 complete [ 7027.718099] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7031.395956] Key type lgssc unregistered [ 7031.761045] LNet: 141640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7031.769656] LNetError: 141640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7031.787570] LNet: Removed LNI 192.168.203.155@tcp [ 7032.498175] Key type .llcrypt unregistered [ 7032.502843] Key type ._llcrypt unregistered [ 7053.926595] Key type ._llcrypt registered [ 7053.932611] Key type .llcrypt registered [ 7054.017384] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7070.031463] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7070.825617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7070.884779] alg: No test for adler32 (adler32-zlib) [ 7071.900896] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 7072.165737] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7073.856533] Key type lgssc registered [ 7075.065554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.671808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7113.709446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.032319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7115.079791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7115.225809] Lustre: lustre-MDT0000: new disk, initializing [ 7115.360089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.381350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7120.169631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7130.740862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.897134] Lustre: 144809:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7130.999069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7131.005482] Lustre: Skipped 1 previous similar message [ 7131.091949] Lustre: lustre-MDT0001: new disk, initializing [ 7131.214690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7131.266031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7131.275184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7136.260636] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7143.892850] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7147.781422] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7155.087116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7155.424330] Lustre: lustre-OST0000: new disk, initializing [ 7155.430083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7155.533060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.413949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7164.527742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7164.536424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7164.699414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7171.169695] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7177.470975] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_module obdecho/obdecho [ 7179.694766] LustreError: 147426:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7179.705101] LustreError: 147426:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7181.529749] LustreError: 147523:0:(echo_client.c:1882:echo_md_handler()) Can not resolve the path : rc = -2 [ 7200.740298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.744107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7200.774713] Lustre: Skipped 1 previous similar message [ 7204.145971] Lustre: server umount lustre-MDT0000 complete [ 7208.520457] LustreError: 144800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582135 with bad export cookie 3191320789446187170 [ 7208.547668] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7209.063901] Lustre: server umount lustre-MDT0001 complete [ 7224.455651] Lustre: server umount lustre-OST0000 complete [ 7246.815145] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7258.036147] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7298.212794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.428800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7298.452561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7298.554819] Lustre: lustre-MDT0000: new disk, initializing [ 7298.660633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.682739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7302.842064] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7314.631476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.766936] Lustre: 151207:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7314.841274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7314.845916] Lustre: Skipped 1 previous similar message [ 7314.929345] Lustre: lustre-MDT0001: new disk, initializing [ 7315.066054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7315.109316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7315.121705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7319.597777] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7327.752701] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7332.155935] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7339.092835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7339.405808] Lustre: lustre-OST0000: new disk, initializing [ 7339.411589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7339.553267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.951308] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7348.814239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7348.837022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7348.908831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7356.086666] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7361.572629] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7361.777511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7367.124356] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7367.320281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7369.696479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7369.696703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.706516] LustreError: Skipped 1 previous similar message [ 7369.722488] Lustre: Skipped 1 previous similar message [ 7369.743388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7369.747048] Lustre: Skipped 2 previous similar messages [ 7374.821176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7374.832306] Lustre: Skipped 1 previous similar message [ 7375.385850] Lustre: server umount lustre-OST0000 complete [ 7379.939903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7379.982868] Lustre: Skipped 1 previous similar message [ 7379.993075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.058531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.064931] Lustre: Skipped 2 previous similar messages [ 7385.748065] Lustre: server umount lustre-MDT0000 complete [ 7389.707978] LustreError: 151200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582316 with bad export cookie 3191320789446188129 [ 7389.719932] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7389.723607] LustreError: 151200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7390.144880] Lustre: server umount lustre-MDT0001 complete [ 7393.824586] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:51:58 (1777582318) [ 7395.617897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7397.476341] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:52:02 (1777582322) [ 7404.982576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.468867] LustreError: 155213:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.487287] LustreError: 155213:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 7405.551675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.844446] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7410.656112] LustreError: 155214:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7415.787661] LustreError: 155213:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.646279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.184197] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7429.353760] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7433.703161] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7440.544354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7440.856840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.862819] Lustre: Skipped 1 previous similar message [ 7446.258814] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7451.881972] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7459.812403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7459.825335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.849596] Lustre: Skipped 1 previous similar message [ 7459.860265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7464.343588] Lustre: server umount lustre-MDT0000 complete [ 7466.468471] LustreError: 156344:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7466.485328] LustreError: 156344:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7468.623071] LustreError: 155193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582395 with bad export cookie 3191320789446189011 [ 7468.624128] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7468.638687] LustreError: 155193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7471.589585] LustreError: 156343:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.593354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.618682] LustreError: 156343:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7471.631941] Lustre: Skipped 2 previous similar messages [ 7475.071201] Lustre: server umount lustre-MDT0001 complete [ 7481.723816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7495.290029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.564606] LustreError: 158717:0:(llog.c:1646:llog_backup()) MGC192.168.203.155@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7495.576683] Lustre: 158717:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.155@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7499.233037] LustreError: 158722:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.270792] LustreError: 158722:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7518.689154] LustreError: 158724:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7518.710317] LustreError: 158724:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 13 previous similar messages [ 7554.112775] LustreError: 158722:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7554.159937] LustreError: 158722:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 26 previous similar messages [ 7556.575288] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7556.587478] Lustre: 158717:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.203.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7597.572138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.028795] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7602.663823] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7610.791234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7614.944887] LustreError: 143030:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005bcccc32 x1863929821263616/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7619.469538] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7620.587979] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7625.015778] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7626.541194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7630.557298] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7632.196032] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7655.258688] Lustre: server umount lustre-MDT0000 complete [ 7656.421380] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.426349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7656.436838] LustreError: 159453:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.436848] LustreError: 159453:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 45 previous similar messages [ 7659.469672] LustreError: 155194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582586 with bad export cookie 3191320789446195465 [ 7659.477411] LustreError: 155194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7659.477740] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.894550] Lustre: server umount lustre-MDT0001 complete [ 7674.518466] Lustre: server umount lustre-OST0000 complete [ 7695.889396] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7705.313695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7743.777761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.037131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7744.088741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7744.216727] Lustre: lustre-MDT0000: new disk, initializing [ 7744.304670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.315475] Lustre: Skipped 1 previous similar message [ 7744.368384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7748.899672] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7760.028413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.142151] Lustre: 163560:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7760.328145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7764.759734] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7772.745930] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7776.787734] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7783.919317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7784.180292] Lustre: lustre-OST0000: new disk, initializing [ 7784.184479] Lustre: Skipped 1 previous similar message [ 7784.193772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7784.203929] Lustre: Skipped 2 previous similar messages [ 7790.273330] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7793.293600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7793.309652] Lustre: Skipped 1 previous similar message [ 7793.322288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7793.449804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7798.312653] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7804.040809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7804.313993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7809.360271] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7809.620894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7814.114938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7814.126924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7814.134414] Lustre: Skipped 2 previous similar messages [ 7814.138880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7814.164948] Lustre: Skipped 8 previous similar messages [ 7817.863660] Lustre: server umount lustre-OST0000 complete [ 7828.631367] Lustre: server umount lustre-MDT0000 complete [ 7829.474373] LustreError: 163572:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.495420] LustreError: 163572:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7832.831182] LustreError: 163554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582759 with bad export cookie 3191320789446212300 [ 7832.836568] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7832.858022] LustreError: 163554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7833.264508] Lustre: server umount lustre-MDT0001 complete [ 7842.445569] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:59:27 (1777582767) [ 7849.592124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.199766] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7866.597548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.075904] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7878.726790] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7883.138684] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7890.228491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7897.814414] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7904.473847] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7914.642810] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:00:39 (1777582839) [ 7920.608568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7920.609545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.625707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7920.644345] Lustre: Skipped 4 previous similar messages [ 7920.666968] Lustre: Skipped 3 previous similar messages [ 7925.354788] Lustre: server umount lustre-OST0000 complete [ 7928.806977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7943.139643] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7943.295402] Lustre: server umount lustre-MDT0000 complete [ 7946.604452] LustreError: 167549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777582873 with bad export cookie 3191320789446213182 [ 7946.615334] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7946.623744] LustreError: 167549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7946.966508] Lustre: server umount lustre-MDT0001 complete [ 7957.821662] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7960.608971] Key type lgssc unregistered [ 7960.905625] LNet: 170952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7960.925868] LNetError: 170952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7960.949216] LNet: Removed LNI 192.168.203.155@tcp [ 7961.775732] Key type .llcrypt unregistered [ 7961.792568] Key type ._llcrypt unregistered [ 7980.380934] Key type ._llcrypt registered [ 7980.382292] Key type .llcrypt registered [ 7980.466756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7981.602688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7981.755534] alg: No test for adler32 (adler32-zlib) [ 7982.843193] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 7983.116971] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7984.831700] Key type lgssc registered [ 7986.421835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7995.403550] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7995.448380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8011.103305] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8016.223185] LustreError: 172179:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 8021.344738] LustreError: 172179:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8021.355765] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8021.361348] LustreError: MGC192.168.203.155@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8021.374059] LustreError: 172179:0:(tgt_mount.c:1838:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8021.388336] LustreError: 172179:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -22 [ 8021.399499] LustreError: 172179:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 8021.493838] Lustre: server umount lustre-OST0000 complete [ 8021.497578] LustreError: 172179:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8031.141820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.932749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8038.562381] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8042.470433] LustreError: 172678:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8042.495944] LustreError: 172678:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8047.588552] LustreError: 172679:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8047.766542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8048.138298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8052.756637] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8060.842354] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8066.908944] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8075.425172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.664637] LustreError: 174268:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8075.675898] Lustre: 174268:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.203.155@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8075.822195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.512982] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8086.655301] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8089.056329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8089.059382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.069271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8093.155936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8093.157428] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.164497] Lustre: Skipped 1 previous similar message [ 8093.186728] Lustre: Skipped 1 previous similar message [ 8098.275566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.281795] Lustre: Skipped 3 previous similar messages [ 8103.391169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8103.398140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8103.412450] Lustre: Skipped 2 previous similar messages [ 8103.569559] Lustre: server umount lustre-MDT0000 complete [ 8107.506291] LustreError: 172659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583034 with bad export cookie 3381445371750228575 [ 8107.508664] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8107.514714] LustreError: 172659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8107.552406] LustreError: 173391:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009e0a2233 x1863930776417024/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8108.047602] Lustre: server umount lustre-MDT0001 complete [ 8122.265697] Lustre: server umount lustre-OST0000 complete [ 8131.860428] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8143.133048] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8182.344909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.610351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8182.649395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8182.712821] Lustre: lustre-MDT0000: new disk, initializing [ 8182.773849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8182.789991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8186.960340] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8198.133133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8198.294355] Lustre: 177650:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8198.378359] LustreError: 177661:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8198.396059] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8198.401757] Lustre: Skipped 1 previous similar message [ 8198.464514] Lustre: lustre-MDT0001: new disk, initializing [ 8198.626084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8198.665747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8198.676801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8203.177826] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8211.018603] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8215.130206] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8223.756061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8224.114803] Lustre: lustre-OST0000: new disk, initializing [ 8224.118865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8224.236808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8231.093430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8231.474866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8231.486737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8231.662801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8240.141498] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8245.764112] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8246.040388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8250.804932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8251.090444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8257.506158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8257.506547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.516206] LustreError: Skipped 1 previous similar message [ 8257.517160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8257.543412] Lustre: Skipped 1 previous similar message [ 8267.232094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8267.368551] Lustre: server umount lustre-OST0000 complete [ 8272.865867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.877753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8272.890959] Lustre: Skipped 1 previous similar message [ 8272.897777] Lustre: Skipped 3 previous similar messages [ 8277.517202] Lustre: server umount lustre-MDT0000 complete [ 8277.988868] LustreError: 179399:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.012036] LustreError: 179399:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8282.309428] LustreError: 177642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583208 with bad export cookie 3381445371750229296 [ 8282.323175] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8282.323395] LustreError: 177642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8282.800732] Lustre: server umount lustre-MDT0001 complete [ 8292.085737] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:06:56 (1777583216) [ 8312.403330] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8315.425556] Key type lgssc unregistered [ 8315.729714] LNet: 182175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8315.738640] LNetError: 182175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8315.799113] LNet: Removed LNI 192.168.203.155@tcp [ 8316.688732] Key type .llcrypt unregistered [ 8316.691698] Key type ._llcrypt unregistered [ 8336.337165] Key type ._llcrypt registered [ 8336.345687] Key type .llcrypt registered [ 8336.526693] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8337.692121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8337.878822] alg: No test for adler32 (adler32-zlib) [ 8339.040617] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 8339.316735] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8341.007189] Key type lgssc registered [ 8342.318077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8350.080964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8350.114570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8351.638493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8355.900753] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8361.960622] LustreError: 183377:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.978567] LustreError: 183377:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8364.751366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.028315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8369.861212] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8375.292185] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8379.700833] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8387.284125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8387.517443] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8387.521179] LustreError: 185014:0:(llog.c:1687:llog_backup()) MGC192.168.203.155@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8387.528624] Lustre: 185014:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.203.155@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8387.676652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.839540] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8399.708712] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8402.402311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.412317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8402.429063] Lustre: Skipped 2 previous similar messages [ 8402.434365] Lustre: Skipped 1 previous similar message [ 8407.525698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8407.536498] Lustre: Skipped 3 previous similar messages [ 8408.452988] Lustre: server umount lustre-MDT0000 complete [ 8411.830305] LustreError: 183356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583338 with bad export cookie 8592949571386251348 [ 8411.833287] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8411.846249] LustreError: 183356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8412.118225] Lustre: server umount lustre-MDT0001 complete [ 8426.114682] Lustre: server umount lustre-OST0000 complete [ 8436.043989] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8446.682721] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8485.678919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.017931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8486.059567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8486.186169] Lustre: lustre-MDT0000: new disk, initializing [ 8486.291209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8486.317673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8490.554530] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8500.480312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.596468] Lustre: 188394:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8500.634892] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8500.638814] Lustre: Skipped 1 previous similar message [ 8500.641524] LustreError: 189158:0:(llog.c:1687:llog_backup()) MGC192.168.203.155@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8500.647542] LustreError: 189158:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8500.652619] Lustre: 189158:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.155@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8500.671486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8500.675703] Lustre: Skipped 1 previous similar message [ 8500.738386] Lustre: lustre-MDT0001: new disk, initializing [ 8500.830049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8500.859442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8500.874605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8505.495972] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8513.442443] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8517.607577] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8523.861757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8524.081917] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8524.085575] Lustre: Skipped 1 previous similar message [ 8524.090542] LustreError: 190111:0:(llog.c:1687:llog_backup()) MGC192.168.203.155@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8524.098268] LustreError: 190111:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8524.106871] Lustre: 190111:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.155@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8524.114540] Lustre: 190111:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8524.132372] Lustre: lustre-OST0000: new disk, initializing [ 8524.137202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8524.201474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8529.452585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8529.466943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8529.579938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8530.028318] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8537.056609] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8541.523362] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8541.702124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8546.170357] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8546.458099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8549.861922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8549.878229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.900257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8554.663157] Lustre: server umount lustre-OST0000 complete [ 8560.613831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8560.630708] Lustre: Skipped 1 previous similar message [ 8560.636766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8560.656835] Lustre: Skipped 3 previous similar messages [ 8565.728281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8565.744027] Lustre: Skipped 1 previous similar message [ 8573.407127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8573.596644] Lustre: server umount lustre-MDT0000 complete [ 8575.972959] LustreError: 188399:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8575.995260] LustreError: 188399:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8577.490517] LustreError: 188386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583504 with bad export cookie 8592949571386252069 [ 8577.503496] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8577.507406] LustreError: 188386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8577.990325] Lustre: server umount lustre-MDT0001 complete [ 8588.141695] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:11:52 (1777583512) [ 8604.862325] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8616.133244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.655986] LustreError: 193096:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.704095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8621.031888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8622.051901] LustreError: 193097:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.172987] LustreError: 193096:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.805248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.698927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8638.824647] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8646.197406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8653.160973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8662.963796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.127979] Lustre: lustre-OST0001: new disk, initializing [ 8663.130978] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8663.207412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8663.217921] Lustre: Skipped 2 previous similar messages [ 8668.915583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8668.933057] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8669.178051] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8672.263327] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8683.875633] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8693.981649] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8701.027330] Lustre: DEBUG MARKER: ADD OST3 [ 8706.217336] loop: module loaded [ 8706.254708] loop0: detected capacity change from 0 to 204800000 [ 8706.355383] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8706.361572] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8706.373210] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8711.695576] loop0: detected capacity change from 0 to 204800000 [ 8713.193677] Lustre: DEBUG MARKER: STOP OST3 [ 8717.513647] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.784567] Lustre: lustre-OST0003: new disk, initializing [ 8717.798955] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8719.536358] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8719.710725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8719.723903] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8719.765468] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8721.571543] LustreError: 197481:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8722.143320] Lustre: Failing over lustre-OST0003 [ 8722.409532] Lustre: server umount lustre-OST0003 complete [ 8724.358077] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8724.573591] LustreError: 194549:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.971800] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8724.982143] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8724.993307] Lustre: Skipped 1 previous similar message [ 8732.881747] Lustre: DEBUG MARKER: START OST3 again [ 8741.925686] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8742.389966] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8742.397548] Lustre: Skipped 1 previous similar message [ 8742.414030] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8743.466183] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8743.673410] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8743.683717] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8748.767348] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8770.400498] Lustre: Failing over lustre-OST0003 [ 8770.669270] Lustre: server umount lustre-OST0003 complete [ 8773.099243] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.117097] LustreError: 195579:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.140165] LustreError: 195579:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 12 previous similar messages [ 8789.990613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8790.001561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.017511] Lustre: Skipped 1 previous similar message [ 8790.026061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8790.031446] Lustre: Skipped 2 previous similar messages [ 8794.844090] Lustre: server umount lustre-MDT0000 complete [ 8797.982621] LustreError: 195352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583724 with bad export cookie 8592949571386252944 [ 8797.987855] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8797.997870] LustreError: 195352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8804.426684] Lustre: server umount lustre-MDT0001 complete [ 8807.968887] Lustre: server umount lustre-OST0000 complete [ 8811.867064] Lustre: server umount lustre-OST0001 complete [ 8818.386439] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8826.086171] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8865.796358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.997905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8866.055188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8866.136964] Lustre: lustre-MDT0000: new disk, initializing [ 8866.288094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8870.838531] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8882.002859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.098357] Lustre: 202734:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8882.154284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8882.158609] Lustre: Skipped 1 previous similar message [ 8882.216309] Lustre: lustre-MDT0001: new disk, initializing [ 8882.350128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8882.356747] Lustre: Skipped 1 previous similar message [ 8882.398949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8882.417560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8887.175848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8895.288773] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8898.806712] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8906.945477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8909.188534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8909.409648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8914.037930] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8921.536569] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8926.094139] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8926.370253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8931.050135] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8931.350407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8934.893594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8934.898199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8934.916769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8934.935063] Lustre: Skipped 6 previous similar messages [ 8934.941265] Lustre: Skipped 8 previous similar messages [ 8939.653431] Lustre: server umount lustre-OST0000 complete [ 8944.096546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8944.116198] LustreError: Skipped 1 previous similar message [ 8949.683843] Lustre: server umount lustre-MDT0000 complete [ 8950.261334] LustreError: 203565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.294983] LustreError: 203565:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 12 previous similar messages [ 8954.044062] LustreError: 202725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777583880 with bad export cookie 8592949571386255247 [ 8954.047040] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8954.058318] LustreError: 202725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8954.285092] Lustre: server umount lustre-MDT0001 complete [ 8958.490642] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:18:02 (1777583882) [ 8976.719338] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8986.308611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9021.855207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.115512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9022.150173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9022.154597] Lustre: Skipped 1 previous similar message [ 9022.273560] Lustre: lustre-MDT0000: new disk, initializing [ 9022.289189] Lustre: Skipped 1 previous similar message [ 9022.456599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9022.467092] Lustre: Skipped 1 previous similar message [ 9027.425644] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9038.717365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.903816] Lustre: 208903:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9039.234405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9043.738456] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9053.069725] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9057.884624] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9065.927214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9068.596314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9074.036521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9083.221179] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9087.856788] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9088.062577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9092.347709] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9092.535614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9099.232696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9099.245438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9099.253452] LustreError: Skipped 1 previous similar message [ 9099.261188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9099.280326] Lustre: Skipped 3 previous similar messages [ 9099.312553] Lustre: Skipped 5 previous similar messages [ 9100.713734] Lustre: server umount lustre-OST0000 complete [ 9111.553846] Lustre: server umount lustre-MDT0000 complete [ 9114.598206] LustreError: 208915:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9114.618796] LustreError: 208915:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9115.611858] LustreError: 208895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584042 with bad export cookie 8592949571386256122 [ 9115.616309] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9115.628074] LustreError: 208895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9115.990197] Lustre: server umount lustre-MDT0001 complete [ 9124.860917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9130.674762] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9139.881757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9140.308599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9140.315248] Lustre: Skipped 5 previous similar messages [ 9144.901738] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9151.625414] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9156.298951] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9162.961873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9169.524592] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9175.576868] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9186.273336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9186.286341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9186.288537] Lustre: Skipped 3 previous similar messages [ 9186.304500] Lustre: Skipped 4 previous similar messages [ 9191.797191] Lustre: server umount lustre-OST0000 complete [ 9196.512336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9196.525460] LustreError: Skipped 1 previous similar message [ 9201.993546] Lustre: server umount lustre-MDT0000 complete [ 9205.912574] LustreError: 213439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584132 with bad export cookie 8592949571386257011 [ 9205.917167] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9205.922171] LustreError: 213439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9206.240778] Lustre: server umount lustre-MDT0001 complete [ 9315.931689] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9318.992274] Key type lgssc unregistered [ 9319.355112] LNet: 216026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9319.371199] LNetError: 216026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9319.400232] LNet: Removed LNI 192.168.203.155@tcp [ 9320.443514] Key type .llcrypt unregistered [ 9320.452381] Key type ._llcrypt unregistered [ 9347.710058] Key type ._llcrypt registered [ 9347.718406] Key type .llcrypt registered [ 9347.861570] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9366.575468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9367.742522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9367.782152] alg: No test for adler32 (adler32-zlib) [ 9369.053960] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [ 9369.308972] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9371.056207] Key type lgssc registered [ 9372.498361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9411.872361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9411.909634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9413.212442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9413.268172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9413.370311] Lustre: lustre-MDT0000: new disk, initializing [ 9413.481166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9413.510988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9418.880841] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9430.005524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9430.131486] Lustre: 219348:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9430.177768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9430.181233] Lustre: Skipped 1 previous similar message [ 9430.248683] Lustre: lustre-MDT0001: new disk, initializing [ 9430.327832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9430.379184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9430.389092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9434.961732] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9442.607593] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9446.773644] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9454.883799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.256673] Lustre: lustre-OST0000: new disk, initializing [ 9455.261233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9455.338339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9460.859488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9460.885490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9461.029316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9462.410187] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9471.150730] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9476.273594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9476.536045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9481.380450] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9481.603808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9486.824026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9486.826201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.862902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9486.866696] Lustre: Skipped 1 previous similar message [ 9490.072759] Lustre: server umount lustre-OST0000 complete [ 9497.061362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9497.067146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9497.069639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9497.074723] Lustre: Skipped 2 previous similar messages [ 9497.081212] LustreError: Skipped 1 previous similar message [ 9500.494457] Lustre: server umount lustre-MDT0000 complete [ 9504.089332] LustreError: 219339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584430 with bad export cookie 2018231284395456975 [ 9504.096752] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9504.104119] LustreError: 219339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9504.424723] Lustre: server umount lustre-MDT0001 complete [ 9523.841243] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9535.960337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9536.395378] LustreError: 223861:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9536.423609] LustreError: 223861:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 9536.490869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9541.602758] LustreError: 223862:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9541.686353] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9546.722816] LustreError: 223861:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9550.293956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.027237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9555.624996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9559.131317] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9565.885378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9566.273534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9572.128919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9580.360223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9580.523596] Lustre: lustre-OST0001: new disk, initializing [ 9580.528255] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9585.922932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9587.792256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9587.803415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9587.913072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9596.861664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9604.039632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9623.976392] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:29:08 (1777584548) [ 9627.619968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9627.629986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9627.654499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9627.665391] Lustre: Skipped 1 previous similar message [ 9632.269134] Lustre: server umount lustre-MDT0000 complete [ 9634.275335] LustreError: 223862:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.300772] LustreError: 223862:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 9635.893064] LustreError: 223842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584562 with bad export cookie 2018231284395457857 [ 9635.894204] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9635.907425] LustreError: 223842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9636.364688] Lustre: server umount lustre-MDT0001 complete [ 9650.665955] Lustre: server umount lustre-OST0000 complete [ 9665.252066] Lustre: server umount lustre-OST0001 complete [ 9671.457980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9681.162887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9717.970687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.314280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9718.352258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9718.443296] Lustre: lustre-MDT0000: new disk, initializing [ 9718.574135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.579198] Lustre: Skipped 1 previous similar message [ 9718.605367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9722.568065] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9733.357176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9733.488874] Lustre: 230040:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9733.597034] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9733.604036] Lustre: Skipped 1 previous similar message [ 9733.726198] Lustre: lustre-MDT0001: new disk, initializing [ 9733.875055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9733.888920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9738.438317] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9746.142115] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9749.741612] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9756.726383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9756.961660] Lustre: lustre-OST0000: new disk, initializing [ 9756.968533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9757.068253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9757.072553] Lustre: Skipped 1 previous similar message [ 9758.102070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9758.117648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9758.294564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9763.687421] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9770.740968] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9775.179610] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9775.395307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9779.669641] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9779.844034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9783.781780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9783.784150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.798070] LustreError: Skipped 1 previous similar message [ 9783.802926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9783.807544] Lustre: Skipped 4 previous similar messages [ 9783.842509] Lustre: Skipped 5 previous similar messages [ 9787.904828] Lustre: server umount lustre-OST0000 complete [ 9794.023159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9794.046913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9794.057857] Lustre: Skipped 1 previous similar message [ 9797.969554] Lustre: server umount lustre-MDT0000 complete [ 9799.138864] LustreError: 230051:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.153375] LustreError: 230051:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9801.556710] LustreError: 230032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584728 with bad export cookie 2018231284395459999 [ 9801.561215] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.572070] LustreError: 230032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9801.981329] Lustre: server umount lustre-MDT0001 complete [ 9810.270709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9810.749763] LustreError: 233850:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.239539] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9823.825488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.190877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9824.198546] Lustre: Skipped 1 previous similar message [ 9828.300330] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9834.189024] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9838.065569] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9845.115605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9851.941136] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9857.706878] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10004.457968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10004.459954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10004.462569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10004.474775] Lustre: Skipped 2 previous similar messages [10009.465620] Lustre: server umount lustre-OST0000 complete [10013.666159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10019.811330] LustreError: 233850:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10019.822989] LustreError: 233850:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10019.939751] Lustre: server umount lustre-MDT0000 complete [10023.907696] LustreError: 233831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777584950 with bad export cookie 2018231284395460881 [10023.912322] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10023.925828] LustreError: 233831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10024.189846] Lustre: server umount lustre-MDT0001 complete [10126.502099] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10129.378939] Key type lgssc unregistered [10129.743786] LNet: 237105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10129.754766] LNetError: 237105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10129.778312] LNet: Removed LNI 192.168.203.155@tcp [10130.727148] Key type .llcrypt unregistered [10130.731502] Key type ._llcrypt unregistered [10156.692643] Key type ._llcrypt registered [10156.696950] Key type .llcrypt registered [10156.807510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10172.229077] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10173.397777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10173.515417] alg: No test for adler32 (adler32-zlib) [10174.559628] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [10174.875911] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10176.543308] Key type lgssc registered [10177.757730] Lustre: Echo OBD driver; http://www.lustre.org/ [10218.216893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10227.078332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10227.089334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.279593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10228.324864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10228.462089] Lustre: lustre-MDT0000: new disk, initializing [10228.609470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10228.631038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10232.825860] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10241.781381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.906332] Lustre: 241135:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10241.958995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10241.962648] Lustre: Skipped 1 previous similar message [10242.022295] Lustre: lustre-MDT0001: new disk, initializing [10242.122285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10242.163871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10242.177280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10246.280902] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10250.533652] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10256.663028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10257.025643] Lustre: lustre-OST0000: new disk, initializing [10257.029655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10257.105770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10258.692522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10258.712688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10258.857820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10262.948934] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10273.419958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10273.587779] Lustre: lustre-OST0001: new disk, initializing [10273.593343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10273.711195] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10280.145250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10283.103794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10283.111050] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10283.224806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10290.165967] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10298.625359] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10314.841265] Lustre: server umount lustre-MDT0000 complete [10318.251683] LustreError: 242740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585244 with bad export cookie 13418071130954184430 [10318.264208] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10318.270572] LustreError: 242740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10318.510298] Lustre: server umount lustre-MDT0001 complete [10335.071293] Lustre: 238664:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777585245/real 1777585245] req@ffff94c68cb01500 x1863933074940928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777585261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10335.097945] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10338.527191] Lustre: 238662:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777585249/real 1777585250] req@ffff94c68cb03480 x1863933074941312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777585265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10338.573774] Lustre: 238662:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10342.445297] Lustre: server umount lustre-OST0000 complete [10344.287245] Lustre: 238664:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777585254/real 1777585254] req@ffff94c786f09880 x1863933074941824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777585270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10344.303801] Lustre: 238664:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10345.374367] Lustre: server umount lustre-OST0001 complete [10357.314394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10365.679747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10372.642529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10381.455128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10400.580189] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10409.130879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10409.147378] Lustre: lustre-MDT0000: reset Object Index mappings [10409.519696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10412.405551] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10414.562614] LustreError: 248563:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.579813] LustreError: 248563:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10418.857611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10418.884193] Lustre: lustre-MDT0001: reset Object Index mappings [10419.220920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.422371] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10424.816312] Lustre: 249668:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10429.791567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.751384] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10435.235735] LustreError: 250023:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.254420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10440.676161] LustreError: 250023:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10441.775636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10442.056994] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10442.062723] Lustre: Skipped 1 previous similar message [10444.165901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10447.898147] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10454.958247] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10458.523452] Lustre: 251507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10469.870387] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10469.872881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10469.882171] Lustre: Skipped 3 previous similar messages [10469.895717] Lustre: Skipped 3 previous similar messages [10470.629748] Lustre: server umount lustre-MDT0000 complete [10474.022248] LustreError: 248542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585400 with bad export cookie 13418071130954191220 [10474.026705] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10474.035084] LustreError: 248542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10474.334117] Lustre: server umount lustre-MDT0001 complete [10488.044376] Lustre: server umount lustre-OST0000 complete [10491.040768] Lustre: 238664:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777585401/real 1777585401] req@ffff94c6831f7480 x1863933074989440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777585417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10491.067410] Lustre: 238664:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10501.608467] Lustre: server umount lustre-OST0001 complete [10518.468483] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:44:03 (1777585443) [10532.445963] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10542.302612] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10572.870482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.078231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10573.130817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10573.193756] Lustre: lustre-MDT0000: new disk, initializing [10573.273766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10573.289530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10577.207546] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10586.839984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.943291] Lustre: 255730:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10586.983955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10586.987461] Lustre: Skipped 1 previous similar message [10587.045416] Lustre: lustre-MDT0001: new disk, initializing [10587.156455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10587.174863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10590.776991] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10597.512596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10600.886675] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10607.131349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10607.343094] Lustre: lustre-OST0000: new disk, initializing [10607.346554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10609.325089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10609.336445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10609.469860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10612.988737] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10620.472287] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10625.160289] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10625.380781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10629.936263] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10630.138558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10635.238212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10635.250064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10635.263283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10635.266780] Lustre: Skipped 2 previous similar messages [10635.281247] Lustre: Skipped 1 previous similar message [10638.393953] Lustre: server umount lustre-OST0000 complete [10643.431302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10643.439198] LustreError: Skipped 1 previous similar message [10643.449649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.467580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10645.472513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10645.483154] Lustre: Skipped 1 previous similar message [10648.638977] Lustre: server umount lustre-MDT0000 complete [10650.594910] LustreError: 256600:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.621239] LustreError: 256600:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10652.337638] LustreError: 255721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585578 with bad export cookie 13418071130954193313 [10652.346877] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10652.353865] LustreError: 255721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10652.760446] Lustre: server umount lustre-MDT0001 complete [10670.073834] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10679.556066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10680.004067] LustreError: 260238:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.079912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10680.088323] Lustre: Skipped 2 previous similar messages [10684.279401] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10689.508373] LustreError: 260239:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.521103] LustreError: 260239:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10693.102612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.347558] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10701.448672] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10707.985864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.290176] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10722.233445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10722.402706] Lustre: lustre-OST0001: new disk, initializing [10722.407032] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10724.437821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10724.453721] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10724.595770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10728.231676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10736.984047] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10740.539102] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10750.840251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10750.905947] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10760.160202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10760.168069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.182823] Lustre: Skipped 1 previous similar message [10760.190613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10763.349668] Lustre: server umount lustre-MDT0000 complete [10766.631656] LustreError: 260219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585693 with bad export cookie 13418071130954194195 [10766.641446] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.642366] LustreError: 260219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10766.820715] LustreError: 260235:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.833401] LustreError: 260235:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10766.936982] Lustre: server umount lustre-MDT0001 complete [10772.856370] Lustre: server umount lustre-OST0000 complete [10776.634350] Lustre: server umount lustre-OST0001 complete [10792.370606] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10802.239757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.606774] LustreError: 265520:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.484457] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10813.708540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10814.150210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10814.158647] Lustre: Skipped 4 previous similar messages [10817.674486] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10820.083762] Lustre: 266627:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10825.779952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10828.151823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10830.944290] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10838.507185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.150885] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10844.159068] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10850.678898] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10853.827401] Lustre: 268471:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10862.806896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10862.907952] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10870.885608] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:49:55 (1777585795) [10885.092612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10885.095825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10885.109125] Lustre: Skipped 8 previous similar messages [10885.124893] Lustre: Skipped 7 previous similar messages [10887.133282] Lustre: server umount lustre-MDT0000 complete [10890.221638] LustreError: 265521:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.235801] LustreError: 265521:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [10890.531777] LustreError: 265501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777585817 with bad export cookie 13418071130954200971 [10890.543544] LustreError: 265501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10890.544873] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10892.936831] Lustre: server umount lustre-MDT0001 complete [10910.692287] Lustre: 238663:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777585821/real 1777585821] req@ffff94c681bd8a80 x1863933075169920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777585837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10910.719603] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10912.930642] Lustre: server umount lustre-OST0000 complete [10922.633452] Lustre: server umount lustre-OST0001 complete [10928.618478] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10937.825979] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10978.903764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10987.668686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10987.897547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10987.923667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10988.010774] Lustre: lustre-MDT0000: new disk, initializing [10988.102268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10991.890623] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11001.680728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11001.825690] Lustre: 272659:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11002.064940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11006.312757] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11016.427274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11018.521538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11022.537561] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11032.350049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.463728] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11034.480962] Lustre: Skipped 1 previous similar message [11034.549342] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11038.083468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11046.247333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11049.206922] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11049.211774] Lustre: Skipped 1 previous similar message [11066.859501] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 17:53:11 (1777585991) [11068.898023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11068.914403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.933472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11074.816569] Lustre: server umount lustre-MDT0000 complete [11075.560528] LustreError: 272669:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.575819] LustreError: 272669:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [11077.867627] LustreError: 272649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586004 with bad export cookie 13418071130954208398 [11077.868692] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11077.880093] LustreError: 272649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11078.166710] Lustre: server umount lustre-MDT0001 complete [11091.834703] Lustre: server umount lustre-OST0000 complete [11106.545467] Lustre: server umount lustre-OST0001 complete [11114.522610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.092577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11115.096373] Lustre: Skipped 6 previous similar messages [11119.050459] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11126.802850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.540599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11135.476262] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11138.328722] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11144.311212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11149.211452] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11154.318722] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11175.376994] loop1: detected capacity change from 0 to 204800000 [11175.575607] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11175.589575] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11175.599348] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11179.850307] loop1: detected capacity change from 0 to 204800000 [11184.935796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11185.028088] LustreError: 277324:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs21074: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11185.042323] LustreError: 277324:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs21074 [11185.048070] LustreError: 277324:0:(mgs_handler.c:517:mgs_target_reg()) fs21074-MDT0001: Error checking target index: rc = -13 [11185.055596] LustreError: fs21074-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11185.064559] LustreError: 280763:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -13 [11185.072497] LustreError: 280763:0:(tgt_mount.c:2082:server_put_super()) no obd fs21074-MDT0001 [11185.076363] LustreError: 280763:0:(tgt_mount.c:114:server_deregister_mount()) fs21074-MDT0001 not registered [11185.107653] Lustre: server umount fs21074-MDT0001 complete [11185.111579] LustreError: 280763:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11192.591650] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.665851] Lustre: Setting parameter fs21074-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21074-MDT0001 [11192.715238] Lustre: srv-fs21074-MDT0001: No data found on store. Initialize space. [11192.722314] Lustre: Skipped 4 previous similar messages [11192.797569] Lustre: fs21074-MDT0001: new disk, initializing [11192.802515] Lustre: Skipped 3 previous similar messages [11192.906043] Lustre: srv-fs21074-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11192.914951] Lustre: 281231:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21074-MDT0001-mdtlov: can't create new llog: rc = -115 [11192.927145] LustreError: 281231:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs21074-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11197.204149] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11203.438521] Lustre: server umount fs21074-MDT0001 complete [11215.955640] loop1: detected capacity change from 0 to 204800000 [11221.068382] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.455442] Lustre: 282782:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21074-MDT0001-mdtlov: can't create new llog: rc = -115 [11221.462428] LustreError: 282782:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs21074-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11225.226407] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11229.332833] Lustre: server umount fs21074-MDT0001 complete [11236.840704] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11239.904300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.920575] Lustre: Skipped 3 previous similar messages [11239.926798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11239.933203] Lustre: Skipped 4 previous similar messages [11252.195986] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11252.413127] Lustre: server umount lustre-MDT0000 complete [11259.149834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11259.226364] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11262.950733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11264.505207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11270.090067] loop1: detected capacity change from 0 to 204800000 [11274.993525] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.474043] Lustre: 285195:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21074-MDT0001-mdtlov: can't create new llog: rc = -115 [11275.484289] LustreError: 285195:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs21074-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11278.916043] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11282.952034] Lustre: server umount fs21074-MDT0001 complete [11305.441294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11305.450755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11305.451387] LustreError: Skipped 2 previous similar messages [11305.470181] Lustre: Skipped 13 previous similar messages [11306.859952] Lustre: server umount lustre-OST0000 complete [11324.383344] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11324.549051] Lustre: server umount lustre-MDT0000 complete [11327.951591] LustreError: 277318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586254 with bad export cookie 13418071130954211079 [11327.958614] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.970505] LustreError: 277318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11328.423135] Lustre: server umount lustre-MDT0001 complete [11337.659596] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11340.131237] Key type lgssc unregistered [11340.376938] LNet: 287506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11340.383144] LNetError: 287506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11340.403278] LNet: Removed LNI 192.168.203.155@tcp [11341.120143] Key type .llcrypt unregistered [11341.125176] Key type ._llcrypt unregistered [11356.728381] Key type ._llcrypt registered [11356.732615] Key type .llcrypt registered [11356.824510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11357.720239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11357.737343] alg: No test for adler32 (adler32-zlib) [11358.779548] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [11358.973491] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11360.671225] Key type lgssc registered [11361.481543] Lustre: Echo OBD driver; http://www.lustre.org/ [11367.914697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11367.931689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11369.349607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11372.925658] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11378.661453] LustreError: 288740:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.668980] LustreError: 288740:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11379.467619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.855076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11383.423929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11388.240993] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11391.109790] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11396.730416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11397.025236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11401.606676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11405.226133] LustreError: 290340:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.356619] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11410.404049] LustreError: 290341:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11413.475960] LustreError: 290339:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11413.497299] LustreError: 290339:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11418.218038] LustreError: 290340:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11428.417880] LustreError: 290339:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11428.438048] LustreError: 290339:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [11431.369370] loop1: detected capacity change from 0 to 204800000 [11431.579613] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11431.593822] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11431.604444] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11435.512888] loop1: detected capacity change from 0 to 204800000 [11439.614762] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11439.725719] Lustre: fs17475-OST0000: new disk, initializing [11439.729503] Lustre: srv-fs17475-OST0000: No data found on store. Initialize space. [11439.785040] Lustre: fs17475-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11444.617423] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11448.897432] LustreError: 290341:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.916311] LustreError: 290341:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 11 previous similar messages [11450.989473] Lustre: server umount fs17475-OST0000 complete [11463.003488] loop1: detected capacity change from 0 to 204800000 [11470.584038] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11471.100932] Lustre: fs17475-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11479.537761] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11484.741394] LustreError: 290848:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.773111] LustreError: 290848:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 20 previous similar messages [11487.814815] loop2: detected capacity change from 0 to 204800000 [11488.113601] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11488.127660] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11488.145891] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11490.298637] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11493.109904] loop2: detected capacity change from 0 to 204800000 [11495.399986] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11497.758372] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.846512] LustreError: fs17475-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11497.854798] LustreError: fs17475-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11497.864089] LustreError: 295204:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -13 [11497.868316] LustreError: 295204:0:(tgt_mount.c:2082:server_put_super()) no obd fs17475-OST0001 [11497.874327] LustreError: 295204:0:(tgt_mount.c:114:server_deregister_mount()) fs17475-OST0001 not registered [11497.899100] Lustre: server umount fs17475-OST0001 complete [11497.901982] LustreError: 295204:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11506.848917] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11506.975345] Lustre: fs17475-OST0001: new disk, initializing [11506.978375] Lustre: srv-fs17475-OST0001: No data found on store. Initialize space. [11507.061871] Lustre: fs17475-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11512.399516] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11519.864453] Lustre: server umount fs17475-OST0000 complete [11528.247718] Lustre: server umount fs17475-OST0001 complete [11539.973128] loop1: detected capacity change from 0 to 204800000 [11545.185254] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11545.341978] Lustre: fs17475-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11550.439113] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11551.310518] LustreError: 290339:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.335209] LustreError: 290339:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 38 previous similar messages [11556.669728] loop2: detected capacity change from 0 to 204800000 [11561.980658] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11567.478828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11575.021244] Lustre: server umount fs17475-OST0000 complete [11582.347840] Lustre: server umount fs17475-OST0001 complete [11592.678761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11592.679350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11592.683041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11592.688245] Lustre: Skipped 1 previous similar message [11597.796213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11597.804024] Lustre: Skipped 2 previous similar messages [11598.102535] Lustre: server umount lustre-OST0000 complete [11602.917496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.930159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11602.933100] Lustre: Skipped 1 previous similar message [11606.881619] Lustre: server umount lustre-MDT0000 complete [11609.656605] LustreError: 288721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586536 with bad export cookie 14018777049684450364 [11609.658866] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11609.667172] LustreError: 288721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11610.025971] Lustre: server umount lustre-MDT0001 complete [11617.866266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11620.127094] Key type lgssc unregistered [11620.344989] LNet: 301620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11620.355772] LNetError: 301620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11620.374801] LNet: Removed LNI 192.168.203.155@tcp [11621.077157] Key type .llcrypt unregistered [11621.080693] Key type ._llcrypt unregistered [11646.174696] Key type ._llcrypt registered [11646.177401] Key type .llcrypt registered [11646.264561] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11657.481722] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11658.239107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11658.353370] alg: No test for adler32 (adler32-zlib) [11659.416964] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [11659.644571] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11661.279274] Key type lgssc registered [11662.201741] Lustre: Echo OBD driver; http://www.lustre.org/ [11690.320618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11690.350599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.604649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11691.627816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11691.722677] Lustre: lustre-MDT0000: new disk, initializing [11691.805048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11691.831163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11695.188941] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11703.664288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.749508] Lustre: 305422:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11703.824929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11703.828646] Lustre: Skipped 1 previous similar message [11703.928546] Lustre: lustre-MDT0001: new disk, initializing [11704.045717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11704.097323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11704.111627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11707.690756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11713.913588] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11716.807601] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11722.625573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11722.887580] Lustre: lustre-OST0000: new disk, initializing [11722.894482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11723.011641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11727.935870] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11728.936217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11728.946625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11729.020135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11734.314888] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11736.047095] Lustre: Mounted lustre-client [11736.782989] LustreError: 307920:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff94c68bf1f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [11736.828451] Lustre: Unmounted lustre-client [11744.224142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11744.228113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11744.228345] LustreError: Skipped 1 previous similar message [11744.243656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11744.255142] Lustre: Skipped 1 previous similar message [11746.282112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11746.289944] Lustre: Skipped 1 previous similar message [11749.743225] Lustre: server umount lustre-OST0000 complete [11755.488598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11755.500925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11755.513390] Lustre: Skipped 1 previous similar message [11755.519382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11756.514805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.842178] Lustre: server umount lustre-MDT0000 complete [11761.657637] LustreError: 306201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586688 with bad export cookie 18213071431056049389 [11761.659335] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11761.672739] LustreError: 306201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11762.030887] Lustre: server umount lustre-MDT0001 complete [11769.777943] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11771.714397] Key type lgssc unregistered [11790.367561] LNet: 309200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11790.379958] LNetError: 309200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11867.173423] LNet: Removed LNI 192.168.203.155@tcp [11867.967620] Key type .llcrypt unregistered [11867.971934] Key type ._llcrypt unregistered [11882.693474] Key type ._llcrypt registered [11882.696944] Key type .llcrypt registered [11882.798533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11896.226549] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11897.166688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11897.189378] alg: No test for adler32 (adler32-zlib) [11898.173736] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [11898.378121] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11900.023446] Key type lgssc registered [11900.900917] Lustre: Echo OBD driver; http://www.lustre.org/ [11924.264229] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:07:29 (1777586849) [11929.264325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11929.276863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.535764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11930.580723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11930.638182] Lustre: lustre-MDT0000: new disk, initializing [11930.719180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11930.741243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11934.037547] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11942.599195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11942.722806] Lustre: 312385:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11942.773509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11942.777582] Lustre: Skipped 1 previous similar message [11942.829922] Lustre: lustre-MDT0001: new disk, initializing [11942.900746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.937787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11942.948054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11946.244471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11952.336401] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11955.250718] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11960.272078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11960.454766] Lustre: lustre-OST0000: new disk, initializing [11960.458295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11960.529549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11964.667293] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11970.094597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11970.109559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11970.176228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11971.046344] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11980.768881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11980.774438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11980.782633] LustreError: Skipped 1 previous similar message [11980.783790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11980.799223] Lustre: Skipped 1 previous similar message [11980.819841] Lustre: Skipped 1 previous similar message [11984.529760] Lustre: server umount lustre-OST0000 complete [11988.961941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11988.976759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11989.005575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.011914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.012038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.034417] Lustre: Skipped 1 previous similar message [11993.691735] Lustre: server umount lustre-MDT0000 complete [11996.677709] LustreError: 312378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777586923 with bad export cookie 11426486115153437909 [11996.681350] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11996.695262] LustreError: 312378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11996.964065] Lustre: server umount lustre-MDT0001 complete [12006.008783] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12008.311279] Key type lgssc unregistered [12008.538551] LNet: 315756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12008.545327] LNetError: 315756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12008.563855] LNet: Removed LNI 192.168.203.155@tcp [12009.190252] Key type .llcrypt unregistered [12009.192338] Key type ._llcrypt unregistered [12030.285649] Key type ._llcrypt registered [12030.287616] Key type .llcrypt registered [12030.384705] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12030.996278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12031.078547] alg: No test for adler32 (adler32-zlib) [12032.047795] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [12032.185919] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12033.855133] Key type lgssc registered [12034.868123] Lustre: Echo OBD driver; http://www.lustre.org/ [12040.723486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12040.741051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12042.157267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.383166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12051.220708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12051.482549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12054.307414] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12058.558594] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12061.439363] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12066.207750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12066.517111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12071.210738] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12075.957626] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12082.278481] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12083.695693] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12087.280049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12087.282391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12087.283482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12087.289722] Lustre: Skipped 1 previous similar message [12087.904127] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [12087.913202] Lustre: Skipped 2 previous similar messages [12092.385371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12092.393146] Lustre: Skipped 2 previous similar messages [12097.504685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12097.517916] Lustre: Skipped 3 previous similar messages [12101.088552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12101.283196] Lustre: server umount lustre-MDT0000 complete [12102.629828] LustreError: 317152:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.649826] LustreError: 317152:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [12103.233923] LustreError: 317148:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12104.404689] LustreError: 317133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777587031 with bad export cookie 14085206523045722899 [12104.417244] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12107.744632] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12107.746783] LustreError: 317146:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.757303] Lustre: Skipped 1 previous similar message [12107.788269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12107.802129] Lustre: Skipped 1 previous similar message [12109.791980] LustreError: 317148:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.806919] LustreError: 317148:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [12110.813816] Lustre: server umount lustre-MDT0001 complete [12114.343744] Lustre: server umount lustre-OST0000 complete [12121.326708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12124.867567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12127.046185] Lustre: 320878:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [12128.702428] Lustre: server umount MGS complete [12135.573343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12135.925840] LustreError: 321449:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.979608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12139.499251] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12146.146199] LustreError: 321450:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.161643] LustreError: 321450:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [12146.378343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12146.621421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12150.324985] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12156.762462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12157.001691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12162.070995] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12166.715624] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12168.227846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12171.403775] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12172.947836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12178.918624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12178.925939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12178.943762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12178.947533] Lustre: Skipped 2 previous similar messages [12181.988467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12181.988885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12183.127215] Lustre: server umount lustre-OST0000 complete [12187.107821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12187.118385] Lustre: Skipped 1 previous similar message [12192.223776] Lustre: server umount lustre-MDT0000 complete [12192.224820] LustreError: 322862:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.248276] LustreError: 322862:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [12195.108501] LustreError: 321429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777587121 with bad export cookie 14085206523045723746 [12195.114674] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12195.130523] LustreError: 321429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12195.409376] Lustre: server umount lustre-MDT0001 complete [12205.281674] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12207.780766] Key type lgssc unregistered [12208.045157] LNet: 324625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12208.052862] LNetError: 324625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12208.074665] LNet: Removed LNI 192.168.203.155@tcp [12208.792842] Key type .llcrypt unregistered [12208.795177] Key type ._llcrypt unregistered [12223.409122] Key type ._llcrypt registered [12223.410726] Key type .llcrypt registered [12223.528182] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12236.756953] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12237.557101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12237.569360] alg: No test for adler32 (adler32-zlib) [12238.528643] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [12238.693110] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12240.335156] Key type lgssc registered [12241.117662] Lustre: Echo OBD driver; http://www.lustre.org/ [12267.471703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12267.491042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12268.685540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12268.712341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12268.794480] Lustre: lustre-MDT0000: new disk, initializing [12268.873852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12268.891387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12272.206325] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12280.355236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12280.457250] Lustre: 327749:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12280.489232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12280.492838] Lustre: Skipped 1 previous similar message [12280.564530] Lustre: lustre-MDT0001: new disk, initializing [12280.647672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12280.680598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12280.688047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12283.936112] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12289.849693] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12292.637071] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12297.269615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12297.452552] Lustre: lustre-OST0000: new disk, initializing [12297.456208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12297.514497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12301.547124] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12307.411049] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12307.495276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12307.502469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12307.591265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12310.670779] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12310.861290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12314.166803] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12314.342447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12317.664734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12317.673897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.685249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12318.183523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12318.186537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12318.186655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12318.215453] Lustre: Skipped 1 previous similar message [12321.961946] Lustre: server umount lustre-OST0000 complete [12326.882504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12326.895459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12326.915419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12333.538310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12333.544401] Lustre: Skipped 3 previous similar messages [12338.656308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12338.665256] Lustre: Skipped 1 previous similar message [12339.167134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12339.319254] Lustre: server umount lustre-MDT0000 complete [12342.095316] LustreError: 328525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777587268 with bad export cookie 13873777823066188413 [12342.098313] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12342.106424] LustreError: 328525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12342.374605] Lustre: server umount lustre-MDT0001 complete [12349.297734] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:14:34 (1777587274) [12361.871308] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12369.154425] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12396.298057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12396.520539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12396.555620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12396.619322] Lustre: lustre-MDT0000: new disk, initializing [12396.676815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12396.688275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12400.188815] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12408.380753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12408.474964] Lustre: 334067:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12408.510224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12408.512810] Lustre: Skipped 1 previous similar message [12408.562192] Lustre: lustre-MDT0001: new disk, initializing [12408.637780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12408.668523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12408.675723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12411.609211] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12417.695131] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12420.734891] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12425.987834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12426.193211] Lustre: lustre-OST0000: new disk, initializing [12426.197306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12426.292416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12430.505115] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12431.936749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12431.944368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12432.015484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12436.388054] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12463.076481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12463.081173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.086529] LustreError: Skipped 1 previous similar message [12463.088615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12463.092573] Lustre: Skipped 2 previous similar messages [12463.109197] Lustre: Skipped 1 previous similar message [12466.570236] Lustre: server umount lustre-OST0000 complete [12470.240743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12470.250070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12475.219516] Lustre: server umount lustre-MDT0000 complete [12477.921526] LustreError: 334060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777587404 with bad export cookie 13873777823066189295 [12477.924326] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12477.929096] LustreError: 334060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12478.197923] Lustre: server umount lustre-MDT0001 complete [12485.563465] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12487.430129] Key type lgssc unregistered [12487.659677] LNet: 337826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12487.666608] LNetError: 337826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12487.683836] LNet: Removed LNI 192.168.203.155@tcp [12488.198407] Key type .llcrypt unregistered [12488.200624] Key type ._llcrypt unregistered [12548.894053] Key type ._llcrypt registered [12548.896975] Key type .llcrypt registered [12548.978114] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12549.156924] Key type .llcrypt unregistered [12549.159828] Key type ._llcrypt unregistered [12599.264050] Key type ._llcrypt registered [12599.267609] Key type .llcrypt registered [12599.352652] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12599.525405] Key type .llcrypt unregistered [12599.528732] Key type ._llcrypt unregistered [12618.027551] Key type ._llcrypt registered [12618.029889] Key type .llcrypt registered [12618.121225] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12618.335414] Key type .llcrypt unregistered [12618.342185] Key type ._llcrypt unregistered [12637.506916] Key type ._llcrypt registered [12637.508537] Key type .llcrypt registered [12637.590629] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12637.786364] Key type .llcrypt unregistered [12637.790208] Key type ._llcrypt unregistered [12684.890904] Key type ._llcrypt registered [12684.892415] Key type .llcrypt registered [12684.950961] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12685.111457] Key type .llcrypt unregistered [12685.114805] Key type ._llcrypt unregistered [12701.665530] Key type ._llcrypt registered [12701.667769] Key type .llcrypt registered [12701.735328] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12701.877364] Key type .llcrypt unregistered [12701.879053] Key type ._llcrypt unregistered [12718.550080] Key type ._llcrypt registered [12718.554744] Key type .llcrypt registered [12718.621591] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12718.791235] Key type .llcrypt unregistered [12718.792703] Key type ._llcrypt unregistered [12735.851962] Key type ._llcrypt registered [12735.856572] Key type .llcrypt registered [12735.948109] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12736.152172] Key type .llcrypt unregistered [12736.154245] Key type ._llcrypt unregistered [12752.831524] Key type ._llcrypt registered [12752.833600] Key type .llcrypt registered [12752.915775] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12753.089695] Key type .llcrypt unregistered [12753.095860] Key type ._llcrypt unregistered [12769.540346] Key type ._llcrypt registered [12769.544405] Key type .llcrypt registered [12769.606423] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12769.772270] Key type .llcrypt unregistered [12769.775756] Key type ._llcrypt unregistered [12805.251247] Key type ._llcrypt registered [12805.252582] Key type .llcrypt registered [12805.318460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12805.500481] Key type .llcrypt unregistered [12805.502271] Key type ._llcrypt unregistered [12823.143829] Key type ._llcrypt registered [12823.145972] Key type .llcrypt registered [12823.211160] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12823.385303] Key type .llcrypt unregistered [12823.386968] Key type ._llcrypt unregistered [12848.652411] Key type ._llcrypt registered [12848.656483] Key type .llcrypt registered [12848.886815] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12849.316206] Key type .llcrypt unregistered [12849.324559] Key type ._llcrypt unregistered [12928.864914] Key type ._llcrypt registered [12928.869398] Key type .llcrypt registered [12929.049804] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12947.153748] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12948.034359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12948.230338] alg: No test for adler32 (adler32-zlib) [12949.399520] Lustre: Lustre: Build Version: 2.17.52_53_gcd13cf3 [12949.684432] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12951.375149] Key type lgssc registered [12952.551341] Lustre: Echo OBD driver; http://www.lustre.org/ [12986.977658] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12987.014577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12988.290425] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12988.323636] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12988.433833] Lustre: tst250fs-MDT0000: new disk, initializing [12988.536708] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12988.559065] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12992.848510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13003.420666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.571795] Lustre: 357587:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13003.624110] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13003.630544] Lustre: Skipped 1 previous similar message [13003.724596] Lustre: tst250fs-MDT0001: new disk, initializing [13003.884226] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13003.935518] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13003.955560] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13008.313403] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13013.034295] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13019.179057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13019.512664] Lustre: tst250fs-OST0000: new disk, initializing [13019.524554] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [13019.532709] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13019.637394] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13024.806146] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13024.822906] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13024.865438] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [13025.514591] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13035.450239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13035.654880] Lustre: tst250fs-OST0001: new disk, initializing [13035.668478] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13035.779379] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13041.375588] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13041.771060] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13041.777193] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13041.881085] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13046.309328] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13048.488244] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13048.493844] Lustre: Skipped 2 previous similar messages [13065.185163] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13065.192936] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13065.208333] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13067.747615] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13067.752030] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13067.767237] Lustre: Skipped 2 previous similar messages [13067.772701] Lustre: Skipped 2 previous similar messages [13071.018337] Lustre: server umount tst250fs-MDT0000 complete [13074.539770] LustreError: 357579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777588001 with bad export cookie 1625102205547323395 [13074.544051] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13074.551055] LustreError: 357579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13074.937690] Lustre: server umount tst250fs-MDT0001 complete [13088.977177] Lustre: server umount tst250fs-OST0000 complete [13103.142995] Lustre: server umount tst250fs-OST0001 complete [13118.218260] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13126.491265] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13157.490104] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:28:02 (1777588082) [13159.426982] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13176.570683] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13188.791767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13221.248957] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12860 sec ========= 18:29:05 (1777588145) [13222.878319] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:29:07 (1777588147) === [13226.313532] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:29:11 (1777588151) === [13264.358712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13267.374147] Key type lgssc unregistered [13267.664582] LNet: 369080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13267.673519] LNetError: 369080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13267.685715] LNet: Removed LNI 192.168.203.155@tcp [13268.471369] Key type .llcrypt unregistered [13268.476945] Key type ._llcrypt unregistered