[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470779927 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002302] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004026] kvm-guest: setup PV IPIs [ 0.007460] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010114] LSM: Security Framework initializing [ 0.011041] Yama: becoming mindful. [ 0.012029] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.020785] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024780] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027028] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028097] *** VALIDATE tmpfs *** [ 0.029399] *** VALIDATE proc *** [ 0.031081] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033241] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.039741] debug: unmapping init [mem 0xffffffff89059000-0xffffffff89060fff] [ 0.041860] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043021] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046007] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048013] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050293] rcu: Hierarchical SRCU implementation. [ 0.052393] smp: Bringing up secondary CPUs ... [ 0.053535] x86: Booting SMP configuration: [ 0.054020] .... node #0, CPUs: #1 #2 #3 [ 0.057202] smp: Brought up 1 node, 4 CPUs [ 0.059017] smpboot: Max logical packages: 1 [ 0.060026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140131] node 0 deferred pages initialised in 77ms [ 0.143147] devtmpfs: initialized [ 0.144276] x86/mm: Memory block size: 128MB [ 0.147560] gcov: version magic: 0x41383552 [ 0.150178] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157320] pinctrl core: initialized pinctrl subsystem [ 0.160172] [ 0.160732] ************************************************************* [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ** ** [ 0.168014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.173011] ** This means that this kernel is built to expose internal ** [ 0.176012] ** IOMMU data structures, which may compromise security on ** [ 0.178011] ** your system. ** [ 0.181019] ** ** [ 0.183011] ** If you see this message and you are not debugging the ** [ 0.186013] ** kernel, report this immediately to your vendor! ** [ 0.188013] ** ** [ 0.191011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193015] ************************************************************* [ 0.196741] NET: Registered protocol family 16 [ 0.199537] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209033] cpuidle: using governor menu [ 0.210573] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213392] PCI: Using configuration type 1 for base access [ 0.215136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230166] cryptd: max_cpu_qlen set to 1000 [ 0.234168] ACPI: Added _OSI(Module Device) [ 0.236014] ACPI: Added _OSI(Processor Device) [ 0.237012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244233] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250369] ACPI: Interpreter enabled [ 0.252057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253011] ACPI: Using IOAPIC for interrupt routing [ 0.255093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258312] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270217] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284543] acpiphp: Slot [2] registered [ 0.286114] acpiphp: Slot [5] registered [ 0.288114] acpiphp: Slot [6] registered [ 0.289130] acpiphp: Slot [7] registered [ 0.290100] acpiphp: Slot [8] registered [ 0.291181] acpiphp: Slot [9] registered [ 0.293083] acpiphp: Slot [10] registered [ 0.294123] acpiphp: Slot [3] registered [ 0.295120] acpiphp: Slot [4] registered [ 0.297090] acpiphp: Slot [11] registered [ 0.298161] acpiphp: Slot [12] registered [ 0.300075] acpiphp: Slot [13] registered [ 0.301062] acpiphp: Slot [14] registered [ 0.302127] acpiphp: Slot [15] registered [ 0.304115] acpiphp: Slot [16] registered [ 0.306102] acpiphp: Slot [17] registered [ 0.307102] acpiphp: Slot [18] registered [ 0.309110] acpiphp: Slot [19] registered [ 0.311117] acpiphp: Slot [20] registered [ 0.312125] acpiphp: Slot [21] registered [ 0.314111] acpiphp: Slot [22] registered [ 0.316109] acpiphp: Slot [23] registered [ 0.317102] acpiphp: Slot [24] registered [ 0.319112] acpiphp: Slot [25] registered [ 0.321104] acpiphp: Slot [26] registered [ 0.323119] acpiphp: Slot [27] registered [ 0.324109] acpiphp: Slot [28] registered [ 0.326169] acpiphp: Slot [29] registered [ 0.328097] acpiphp: Slot [30] registered [ 0.329105] acpiphp: Slot [31] registered [ 0.331063] PCI host bridge to bus 0000:00 [ 0.333020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352967] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356168] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364724] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369462] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377467] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380449] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384716] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388689] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405799] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413427] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.466902] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.502271] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.540017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550111] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.559026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.571014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.590016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.600564] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.621027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.636021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.645491] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.651015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.658019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.687180] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.689478] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.693486] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.696451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.699303] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705036] iommu: Default domain type: Passthrough [ 0.706410] SCSI subsystem initialized [ 0.708165] ACPI: bus type USB registered [ 0.710155] usbcore: registered new interface driver usbfs [ 0.712070] usbcore: registered new interface driver hub [ 0.714082] usbcore: registered new device driver usb [ 0.716188] pps_core: LinuxPPS API ver. 1 registered [ 0.718012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.722087] PTP clock support registered [ 0.725035] EDAC MC: Ver: 3.0.0 [ 0.727307] PCI: Using ACPI for IRQ routing [ 0.728573] NetLabel: Initializing [ 0.730016] NetLabel: domain hash size = 128 [ 0.733014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736092] NetLabel: unlabeled traffic allowed by default [ 0.739220] vgaarb: loaded [ 0.741280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.743020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.749384] clocksource: Switched to clocksource kvm-clock [ 0.859464] VFS: Disk quotas dquot_6.6.0 [ 0.861702] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.864871] *** VALIDATE ramfs *** [ 0.866155] *** VALIDATE hugetlbfs *** [ 0.867262] pnp: PnP ACPI init [ 0.868860] pnp: PnP ACPI: found 6 devices [ 0.885061] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.888717] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.891027] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.893431] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.895598] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.897678] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.900856] NET: Registered protocol family 2 [ 0.902952] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.907787] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.911580] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.916975] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.920792] TCP: Hash tables configured (established 65536 bind 65536) [ 0.924669] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.928278] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.933338] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.936791] NET: Registered protocol family 1 [ 0.940656] RPC: Registered named UNIX socket transport module. [ 0.943021] RPC: Registered udp transport module. [ 0.945080] RPC: Registered tcp transport module. [ 0.946990] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.949717] NET: Registered protocol family 44 [ 0.951686] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.954397] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.956937] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.959859] PCI: CLS 0 bytes, default 64 [ 0.961699] Unpacking initramfs... [ 2.369661] debug: unmapping init [mem 0xffff9b3bbcc54000-0xffff9b3bbffbffff] [ 2.375334] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.378876] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.382988] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.890782] Initialise system trusted keyrings [ 2.892902] Key type blacklist registered [ 2.895197] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.905657] zbud: loaded [ 2.909176] *** VALIDATE nfs *** [ 2.910513] *** VALIDATE nfs4 *** [ 2.912659] pstore: using deflate compression [ 2.919478] Platform Keyring initialized [ 3.031575] NET: Registered protocol family 38 [ 3.033407] Key type asymmetric registered [ 3.035166] Asymmetric key parser 'x509' registered [ 3.037197] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.040680] io scheduler mq-deadline registered [ 3.042447] io scheduler kyber registered [ 3.044319] io scheduler bfq registered [ 3.046634] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.049800] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.054279] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.057977] ACPI: Power Button [PWRF] [ 3.065098] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.072455] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.087923] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.103930] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.136225] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.167340] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.195974] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.201326] Non-volatile memory driver v1.3 [ 3.203397] Linux agpgart interface v0.103 [ 3.244229] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.247708] vda: detected capacity change from 0 to 75071488 [ 3.263789] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.266342] vdb: detected capacity change from 0 to 1073741824 [ 3.283198] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.286022] vdc: detected capacity change from 0 to 2621440000 [ 3.302405] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.306226] vdd: detected capacity change from 0 to 2621440000 [ 3.322026] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.325347] vde: detected capacity change from 0 to 4294967296 [ 3.346523] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.351315] vdf: detected capacity change from 0 to 4294967296 [ 3.362768] libphy: Fixed MDIO Bus: probed [ 3.366680] usbcore: registered new interface driver usbserial_generic [ 3.368798] usbserial: USB Serial support registered for generic [ 3.370862] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.374667] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.377066] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.379789] mousedev: PS/2 mouse device common for all mice [ 3.382890] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.386443] rtc_cmos 00:05: RTC can wake from S4 [ 3.389841] rtc_cmos 00:05: registered as rtc0 [ 3.391458] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.392635] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.399184] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.400321] intel_pstate: CPU model not supported [ 3.406570] hid: raw HID events driver (C) Jiri Kosina [ 3.408616] usbcore: registered new interface driver usbhid [ 3.410220] usbhid: USB HID core driver [ 3.411922] drop_monitor: Initializing network drop monitor service [ 3.413974] Initializing XFRM netlink socket [ 3.416228] NET: Registered protocol family 10 [ 3.418942] Segment Routing with IPv6 [ 3.420514] NET: Registered protocol family 17 [ 3.422899] mpls_gso: MPLS GSO support [ 3.428495] RAS: Correctable Errors collector initialized. [ 3.431029] AVX version of gcm_enc/dec engaged. [ 3.433333] AES CTR mode by8 optimization enabled [ 3.511646] sched_clock: Marking stable (3511623511, 0)->(4399122028, -887498517) [ 3.516973] registered taskstats version 1 [ 3.519307] Loading compiled-in X.509 certificates [ 3.522276] zswap: loaded using pool lzo/zbud [ 3.547169] Key type big_key registered [ 3.559688] Key type encrypted registered [ 3.561535] ima: No TPM chip found, activating TPM-bypass! [ 3.564095] ima: Allocated hash algorithm: sha1 [ 3.565939] ima: No architecture policies found [ 3.567807] evm: Initialising EVM extended attributes: [ 3.569732] evm: security.selinux [ 3.570573] evm: security.ima [ 3.571382] evm: security.capability [ 3.572417] evm: HMAC attrs: 0x1 [ 3.574807] rtc_cmos 00:05: setting system clock to 2026-08-27 11:53:11 UTC (1787831591) [ 3.580759] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 3.583993] debug: unmapping init [mem 0xffffffff88d82000-0xffffffff89058fff] [ 3.590066] Write protecting the kernel read-only data: 28672k [ 3.593163] debug: unmapping init [mem 0xffffffff87403000-0xffffffff875fffff] [ 3.596445] debug: unmapping init [mem 0xffffffff87d14000-0xffffffff87dfffff] [ 3.632974] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.644113] systemd[1]: Detected virtualization kvm. [ 3.646051] systemd[1]: Detected architecture x86-64. [ 3.647935] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.670461] systemd[1]: No hostname configured. [ 3.672123] systemd[1]: Set hostname to . [ 3.673531] random: systemd: uninitialized urandom read (16 bytes read) [ 3.676065] systemd[1]: Initializing machine ID from random generator. [ 3.714338] random: ln: uninitialized urandom read (6 bytes read) [ 3.802499] random: systemd: uninitialized urandom read (16 bytes read) [ 3.805401] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.809385] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.814207] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.412637] device-mapper: uevent: version 1.0.3 [ 4.414933] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.056028] virtio_net virtio0 ens2: renamed from eth0 [ 5.065722] random: fast init done [ 5.142973] scsi host0: ata_piix [ 5.175708] scsi host1: ata_piix [ 5.177503] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.179721] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.752452] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.078343] random: crng init done [ 10.080113] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.412220] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.462192] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.719899] SELinux: Disabled at runtime. [ 11.783180] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.794220] systemd[1]: Detected virtualization kvm. [ 11.795634] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.301300] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.304024] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.309651] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.314384] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.316632] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.322215] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.328569] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 12.353697] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.733482] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.939776] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.998824] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.115824] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.129613] EDAC sbridge: Ver: 1.1.2 [ 14.761380] Key type dns_resolver registered [ 15.057040] NFS: Registering the id_resolver key type [ 15.058517] Key type id_resolver registered [ 15.059733] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg344-server login: [ 40.135765] libcfs: loading out-of-tree module taints kernel. [ 40.154949] Key type ._llcrypt registered [ 40.156778] Key type .llcrypt registered [ 40.207403] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 55.606680] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 57.083339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.104191] alg: No test for adler32 (adler32-zlib) [ 58.459862] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 59.348626] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 61.120692] Key type lgssc registered [ 62.753826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.256439] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.151734] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 137.426923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.472201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.899262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.959946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.079420] Lustre: lustre-MDT0000: new disk, initializing [ 139.255857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.310486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.205614] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.187614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.285417] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.312151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.315408] Lustre: Skipped 1 previous similar message [ 159.370798] Lustre: lustre-MDT0001: new disk, initializing [ 159.417504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.455978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.485817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.009822] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.785966] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 177.964490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.391525] Lustre: lustre-OST0000: new disk, initializing [ 178.400492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 178.407629] Lustre: 8441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 178.474258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.333246] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.442855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.460106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.585049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.078844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.215848] Lustre: lustre-OST0001: new disk, initializing [ 199.222747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 199.229938] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.284053] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.287688] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.388804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.405607] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.506770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 217.061083] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.826169] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 230.533830] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 235.595300] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 240.507921] Lustre: DEBUG MARKER: Client: 2.17.57.60 [ 243.172523] Lustre: DEBUG MARKER: MDS: 2.17.57.60 [ 245.868471] Lustre: DEBUG MARKER: OSS: 2.17.57.60 [ 247.755339] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:57:14 EDT 2026 [ 267.341581] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 268.781959] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.717828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.726542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.747991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.738517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.739143] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.751539] Lustre: Skipped 1 previous similar message [ 288.783747] Lustre: Skipped 1 previous similar message [ 291.286462] Lustre: server umount lustre-MDT0000 complete [ 295.121331] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831883 with bad export cookie 3016384621648497654 [ 295.127358] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.141383] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 298.976503] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.979702] LustreError: 9522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.979880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.979884] Lustre: Skipped 2 previous similar messages [ 298.998363] Lustre: Skipped 2 previous similar messages [ 299.030642] LustreError: 9522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 301.034726] LustreError: 6511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.038715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.080868] Lustre: Skipped 2 previous similar messages [ 301.642642] Lustre: server umount lustre-MDT0001 complete [ 305.516232] Lustre: server umount lustre-OST0000 complete [ 309.690339] Lustre: server umount lustre-OST0001 complete [ 316.898302] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 326.439302] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 346.239045] hrtimer: interrupt took 2100746 ns [ 362.129820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.415965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 362.472197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 362.601716] Lustre: lustre-MDT0000: new disk, initializing [ 362.721034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.741958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.283989] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 377.588057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.771879] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 377.893500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 377.900112] Lustre: Skipped 1 previous similar message [ 378.011643] Lustre: lustre-MDT0001: new disk, initializing [ 378.154929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.224663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 378.242235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 382.273947] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 393.429986] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.528598] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.611105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 406.960233] Lustre: lustre-OST0000: new disk, initializing [ 406.963861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.971486] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 407.046438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.909646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 408.932649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.078976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 413.401062] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 422.408985] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 426.707221] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.959068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 431.254761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 431.408825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 434.656587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.667549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.672321] LustreError: Skipped 1 previous similar message [ 434.675578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 434.688904] Lustre: Skipped 1 previous similar message [ 439.343701] Lustre: server umount lustre-OST0000 complete [ 444.899395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.906272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.915204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.927386] Lustre: Skipped 2 previous similar messages [ 448.547182] Lustre: server umount lustre-MDT0000 complete [ 450.019235] LustreError: 15075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.040060] LustreError: 15075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 451.929177] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832039 with bad export cookie 3016384621648499187 [ 451.935154] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 451.939049] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 452.214789] Lustre: server umount lustre-MDT0001 complete [ 458.008411] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:00:44 (1787832044) [ 476.263994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.798505] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.876059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.822971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 482.282165] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.333885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.099518] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 502.322671] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.706315] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.887043] Lustre: Failing over lustre-MDT0000 [ 510.133697] Lustre: server umount lustre-MDT0000 complete [ 511.462698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 511.464744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.473833] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.503615] Lustre: Skipped 2 previous similar messages [ 511.522333] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 527.905340] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832099/real 1787832099] req@ffff9b3c3d785a40 x1874677357241728/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787832115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 527.939379] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.959020] LustreError: 21164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.979801] LustreError: 21164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 529.153627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.080787] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3b032361c0 x1874677357244800/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 542.176497] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 543.716212] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 543.725303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 543.754949] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 551.557804] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 559.073520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.079370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 559.103531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.125391] Lustre: Skipped 1 previous similar message [ 570.335271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 570.471770] Lustre: server umount lustre-MDT0000 complete [ 573.785245] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832161 with bad export cookie 3016384621648500601 [ 573.792329] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.797078] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 574.122739] Lustre: server umount lustre-MDT0001 complete [ 582.701447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.097968] LustreError: 23264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.136259] LustreError: 23264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 583.200718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.216944] Lustre: Skipped 2 previous similar messages [ 587.560526] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 596.401242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.914955] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 607.765370] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 613.303287] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 615.412382] Lustre: Failing over lustre-MDT0000 [ 615.733150] Lustre: server umount lustre-MDT0000 complete [ 617.442735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 617.455052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.478898] Lustre: Skipped 1 previous similar message [ 634.203136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.336105] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.605240] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 638.174506] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 639.969317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 639.970743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 639.991717] Lustre: Skipped 1 previous similar message [ 640.010091] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 646.408859] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 650.209656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.219651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.222460] Lustre: Skipped 6 previous similar messages [ 656.033432] Lustre: server umount lustre-MDT0000 complete [ 659.173410] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832247 with bad export cookie 3016384621648501539 [ 659.177304] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.184866] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 659.515800] Lustre: server umount lustre-MDT0001 complete [ 667.207695] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:04:14 (1787832254) [ 680.999285] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 689.971729] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 728.266913] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 739.336287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.587671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 739.625771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 739.702622] Lustre: lustre-MDT0000: new disk, initializing [ 739.788405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.792673] Lustre: Skipped 2 previous similar messages [ 739.807507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 744.284706] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 754.052268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.167867] Lustre: 29637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 754.456767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 758.105859] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 766.042374] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 771.695496] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 777.480536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.689123] Lustre: 31603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 778.874888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 778.916401] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 778.971231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 778.995744] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 783.535820] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 791.874640] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 796.967905] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 820.221591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 820.226678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.232133] LustreError: Skipped 1 previous similar message [ 820.236541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 820.253137] Lustre: Skipped 4 previous similar messages [ 820.282847] Lustre: Skipped 3 previous similar messages [ 826.225190] Lustre: server umount lustre-OST0000 complete [ 836.942045] LustreError: 32830:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9b3c2fcd3500) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 837.047647] Lustre: server umount lustre-MDT0000 complete [ 840.673655] LustreError: 29647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.700239] LustreError: 29647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 841.138054] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832429 with bad export cookie 3016384621648501945 [ 841.147205] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.166550] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 841.595713] Lustre: server umount lustre-MDT0001 complete [ 853.246384] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 855.743245] Key type lgssc unregistered [ 855.991893] LNet: 33585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.007345] LNetError: 33585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.030895] LNet: Removed LNI 192.168.203.144@tcp [ 856.880145] Key type .llcrypt unregistered [ 856.887051] Key type ._llcrypt unregistered [ 879.817906] Key type ._llcrypt registered [ 879.820381] Key type .llcrypt registered [ 879.940317] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 881.017902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.041820] alg: No test for adler32 (adler32-zlib) [ 882.033129] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 882.271707] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 883.943142] Key type lgssc registered [ 885.091700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.353911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.375966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.690852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.119690] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 903.142505] LustreError: 34998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.163278] LustreError: 34998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 904.856989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.223848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 908.724460] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 916.470330] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 923.699716] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 931.089548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.525036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.274899] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 938.689688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 944.565111] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 950.959259] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 962.058470] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:09:08 (1787832548) [ 963.985449] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 965.573287] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 971.564485] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:09:18 (1787832558) [ 979.877238] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:09:26 (1787832566) [ 981.039752] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 988.019284] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 988.024480] Lustre: Skipped 2 previous similar messages [ 989.739504] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:09:36 (1787832576) [ 993.497416] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 993.501451] Lustre: Skipped 3 previous similar messages [ 1009.709682] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1009.725411] Lustre: Skipped 6 previous similar messages [ 1012.703541] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:09:59 (1787832599) [ 1015.339288] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1018.783948] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1027.640131] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:10:14 (1787832614) [ 1029.601726] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1029.606753] Lustre: Skipped 3 previous similar messages [ 1041.753754] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:10:28 (1787832628) [ 1056.738715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1056.747168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.769494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.788197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.789239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.796906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.911079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.920468] Lustre: Skipped 2 previous similar messages [ 1069.024553] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1069.036910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.041381] Lustre: Skipped 1 previous similar message [ 1069.202403] Lustre: server umount lustre-OST0000 complete [ 1074.146497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1074.150212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.154326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.162273] Lustre: Skipped 1 previous similar message [ 1074.172826] Lustre: Skipped 1 previous similar message [ 1079.158084] Lustre: server umount lustre-MDT0000 complete [ 1079.264831] LustreError: 34994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.281167] LustreError: 34994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1082.641586] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832670 with bad export cookie 17068882577330206159 [ 1082.648189] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1082.651669] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1082.931487] Lustre: server umount lustre-MDT0001 complete [ 1093.727368] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1096.831803] Key type lgssc unregistered [ 1097.163993] LNet: 42846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.187337] LNetError: 42846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.206892] LNet: Removed LNI 192.168.203.144@tcp [ 1098.203604] Key type .llcrypt unregistered [ 1098.206570] Key type ._llcrypt unregistered [ 1117.858338] Key type ._llcrypt registered [ 1117.864039] Key type .llcrypt registered [ 1118.025166] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1135.754195] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1136.647122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.707121] alg: No test for adler32 (adler32-zlib) [ 1137.735226] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1137.950995] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1139.623313] Key type lgssc registered [ 1140.806614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.829904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1178.856429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.028084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1180.069201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1180.176848] Lustre: lustre-MDT0000: new disk, initializing [ 1180.265477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.283324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1185.089679] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1196.067348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.199925] Lustre: 46003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1196.264753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1196.268684] Lustre: Skipped 1 previous similar message [ 1196.343491] Lustre: lustre-MDT0001: new disk, initializing [ 1196.445178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1196.489498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1196.505969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1200.544685] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1209.807274] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.387104] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1224.707386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.094485] Lustre: lustre-OST0000: new disk, initializing [ 1225.110917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1225.131578] Lustre: 47970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1225.254251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.071262] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1234.029713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1234.046439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1234.168197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1242.407634] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1246.692418] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1246.923321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1251.390973] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1251.673541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1254.889408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.898850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1254.908648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1254.917146] LustreError: Skipped 1 previous similar message [ 1254.945587] Lustre: Skipped 1 previous similar message [ 1260.000843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1260.009131] Lustre: Skipped 1 previous similar message [ 1265.120620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.128406] Lustre: Skipped 1 previous similar message [ 1268.191268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1268.380069] Lustre: server umount lustre-OST0000 complete [ 1273.320406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1273.326109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.335482] Lustre: Skipped 1 previous similar message [ 1273.340413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.363854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.840602] Lustre: server umount lustre-MDT0000 complete [ 1282.445327] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832870 with bad export cookie 17322688610590578855 [ 1282.452594] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.458536] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1282.730818] Lustre: server umount lustre-MDT0001 complete [ 1292.803600] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:14:39 (1787832879) [ 1300.176707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.618227] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.634722] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1300.682428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.514892] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1306.090137] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.197685] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.703593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.138936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1319.272775] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1326.453672] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1332.169408] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1339.024625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.386204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1345.455830] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1353.325646] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1357.929847] LustreError: 52959:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1359.006447] Lustre: Modifying parameter general.timeout=1 in log params [ 1360.908337] Lustre: Modifying parameter general.timeout=3 in log params [ 1360.911600] Lustre: Skipped 1 previous similar message [ 1363.209914] Lustre: Modifying parameter general.timeout=5 in log params [ 1363.213151] Lustre: Skipped 1 previous similar message [ 1367.612762] Lustre: Modifying parameter general.timeout=10 in log params [ 1367.615522] Lustre: Skipped 4 previous similar messages [ 1376.155488] Lustre: Modifying parameter general.timeout=18 in log params [ 1376.161122] Lustre: Skipped 7 previous similar messages [ 1392.163823] Lustre: Modifying parameter general.timeout=36 in log params [ 1392.167492] Lustre: Skipped 17 previous similar messages [ 1424.922162] Lustre: Modifying parameter general.timeout=72 in log params [ 1424.933681] Lustre: Skipped 35 previous similar messages [ 1497.225885] Lustre: Disabling parameter general.timeout= in log params [ 1497.232506] Lustre: Skipped 57 previous similar messages [ 1501.173198] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:18:07 (1787833087) [ 1504.049671] Lustre: 59920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1505.821516] Lustre: 60018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1509.362399] Lustre: 60216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1509.373403] Lustre: 60216:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1541.030667] Lustre: 61792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1541.049497] Lustre: 61792:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1541.082026] LustreError: 61792:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1545.357954] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:18:51 (1787833131) [ 1561.056975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1561.069043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.112590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.135903] Lustre: Skipped 2 previous similar messages [ 1567.431382] Lustre: server umount lustre-MDT0000 complete [ 1568.225831] LustreError: 51864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.239504] LustreError: 51864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1571.971882] LustreError: 57219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833159 with bad export cookie 17322688610590579744 [ 1571.987460] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.989262] LustreError: 57219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1572.436769] Lustre: server umount lustre-MDT0001 complete [ 1587.330397] Lustre: server umount lustre-OST0000 complete [ 1615.883346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.086614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1616.317967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.520942] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1630.993837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.164934] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1631.222986] Lustre: 64325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1636.881317] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1645.961447] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1652.659203] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1660.201096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.345631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1660.643674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.656309] Lustre: Skipped 1 previous similar message [ 1668.080364] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1677.439576] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.555592] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1679.564912] Lustre: Skipped 23 previous similar messages [ 1679.616889] Lustre: 66892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1691.105144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1691.113916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.142571] Lustre: Skipped 2 previous similar messages [ 1691.152353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.158232] Lustre: Skipped 3 previous similar messages [ 1694.056969] Lustre: server umount lustre-OST0000 complete [ 1701.865411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.881711] Lustre: Skipped 2 previous similar messages [ 1704.995554] Lustre: server umount lustre-MDT0000 complete [ 1706.977288] LustreError: 64335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.005487] LustreError: 64335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1708.880456] LustreError: 64315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833296 with bad export cookie 17322688610590582901 [ 1708.882159] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.894441] LustreError: 64315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1709.293633] Lustre: server umount lustre-MDT0001 complete [ 1725.121358] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1729.299811] Key type lgssc unregistered [ 1729.629215] LNet: 68102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.639441] LNetError: 68102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.666060] LNet: Removed LNI 192.168.203.144@tcp [ 1730.595802] Key type .llcrypt unregistered [ 1730.598327] Key type ._llcrypt unregistered [ 1758.027196] Key type ._llcrypt registered [ 1758.032292] Key type .llcrypt registered [ 1758.299560] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1759.801710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.830469] alg: No test for adler32 (adler32-zlib) [ 1761.116878] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1761.490949] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1763.215400] Key type lgssc registered [ 1765.498171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.982784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1780.003244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.870725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.504817] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1791.973406] LustreError: 69517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.005722] LustreError: 69517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1795.862448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.651716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.269948] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1811.326633] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1817.542541] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1824.561630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.996852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.441092] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1840.310669] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.524992] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1862.116136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.118219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.136969] Lustre: Skipped 1 previous similar message [ 1862.151214] Lustre: Skipped 2 previous similar messages [ 1867.247542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.262286] Lustre: Skipped 2 previous similar messages [ 1867.626500] Lustre: server umount lustre-MDT0000 complete [ 1871.697102] LustreError: 69497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833459 with bad export cookie 14028572705349823193 [ 1871.700821] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.704479] LustreError: 69497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1872.006274] Lustre: server umount lustre-MDT0001 complete [ 1887.124738] Lustre: server umount lustre-OST0000 complete [ 1918.623833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.926787] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1918.950110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1919.197461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.632438] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1933.736474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.925055] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1933.986896] Lustre: 74393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1934.610302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1939.161889] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1947.814855] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1955.614343] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1964.477719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1964.641118] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1964.976478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.465471] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1981.272366] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1983.679970] Lustre: 76971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1986.157224] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1994.721060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.737674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.762570] Lustre: Skipped 1 previous similar message [ 1994.774502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.865370] Lustre: server umount lustre-OST0000 complete [ 2004.965772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.976482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.981076] Lustre: Skipped 2 previous similar messages [ 2004.990077] Lustre: Skipped 1 previous similar message [ 2010.083431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.100290] Lustre: Skipped 1 previous similar message [ 2010.861360] Lustre: server umount lustre-MDT0000 complete [ 2014.178797] LustreError: 74405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.207767] LustreError: 74405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2015.292371] LustreError: 74386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833603 with bad export cookie 14028572705349824040 [ 2015.299117] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.310555] LustreError: 74386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2015.859682] Lustre: server umount lustre-MDT0001 complete [ 2030.614583] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2034.032415] Key type lgssc unregistered [ 2034.447230] LNet: 78187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.456941] LNetError: 78187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.483278] LNet: Removed LNI 192.168.203.144@tcp [ 2035.703883] Key type .llcrypt unregistered [ 2035.708340] Key type ._llcrypt unregistered [ 2061.601362] Key type ._llcrypt registered [ 2061.602958] Key type .llcrypt registered [ 2061.707369] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2062.793657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.867067] alg: No test for adler32 (adler32-zlib) [ 2063.998279] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2064.304346] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2065.983152] Key type lgssc registered [ 2067.225333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2076.246084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2076.278673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.907136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.191512] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2087.409874] LustreError: 79600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.436412] LustreError: 79600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2092.529422] LustreError: 79601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.698561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.495881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2101.194834] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2109.869825] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2116.411191] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2123.997775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.789509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.038205] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2142.116238] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2153.133893] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2154.173358] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2155.466061] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2157.941306] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2157.955042] Lustre: Skipped 1 previous similar message [ 2162.441719] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2162.452640] Lustre: Skipped 3 previous similar messages [ 2170.766262] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2170.777624] Lustre: Skipped 6 previous similar messages [ 2187.578703] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2187.582679] Lustre: Skipped 15 previous similar messages [ 2220.265972] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2220.270769] Lustre: Skipped 32 previous similar messages [ 2266.756056] Lustre: 87719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2269.383307] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:30:55 (1787833855) [ 2271.477207] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2273.385067] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:30:59 (1787833859) [ 2278.368572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.375294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.401694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.443247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.445855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.451431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.478599] Lustre: Skipped 1 previous similar message [ 2283.768552] Lustre: server umount lustre-OST0000 complete [ 2289.632144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.650207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.667093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.282775] Lustre: server umount lustre-MDT0000 complete [ 2295.781966] LustreError: 79601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.802533] LustreError: 79601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2298.638053] LustreError: 79582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833886 with bad export cookie 18221685841469620510 [ 2298.651757] LustreError: 79582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2298.655827] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.006811] Lustre: server umount lustre-MDT0001 complete [ 2311.383929] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2314.432426] Key type lgssc unregistered [ 2314.697380] LNet: 88971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2314.709440] LNetError: 88971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.724413] LNet: Removed LNI 192.168.203.144@tcp [ 2315.562556] Key type .llcrypt unregistered [ 2315.565253] Key type ._llcrypt unregistered [ 2320.942213] Key type ._llcrypt registered [ 2320.943889] Key type .llcrypt registered [ 2321.117953] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2326.477176] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules [ 2327.972696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.001585] alg: No test for adler32 (adler32-zlib) [ 2328.949514] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2329.302636] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2331.008105] Key type lgssc registered [ 2332.091868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2333.234623] LustreError: 90222:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2351.107795] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2355.838709] LustreError: 90222:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2360.016857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2360.061291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.500472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.813825] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2372.066246] LustreError: 91247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.078556] LustreError: 91247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2375.005940] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:32:41 (1787833961) [ 2377.185643] LustreError: 91246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.601853] LustreError: 91278:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2379.611384] LustreError: 91278:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2379.622089] LustreError: 91278:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2380.036822] Lustre: server umount lustre-MDT0000 complete [ 2398.319446] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2401.661517] Key type lgssc unregistered [ 2402.021090] LNet: 92774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.035148] LNetError: 92774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2402.067333] LNet: Removed LNI 192.168.203.144@tcp [ 2403.023422] Key type .llcrypt unregistered [ 2403.028963] Key type ._llcrypt unregistered [ 2421.724382] Key type ._llcrypt registered [ 2421.730272] Key type .llcrypt registered [ 2421.822280] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2423.055307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2423.114120] alg: No test for adler32 (adler32-zlib) [ 2424.238655] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2424.540218] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2426.306382] Key type lgssc registered [ 2427.725647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2436.420428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2436.434343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.129279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.263959] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2447.333227] LustreError: 93992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.352082] LustreError: 93992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2451.978592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.506497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2456.778684] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2464.046829] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2470.239277] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2477.614767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2478.220708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.140173] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2493.817575] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2544.630506] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:35:31 (1787834131) [ 2546.871664] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2549.129815] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:35:35 (1787834135) [ 2554.338330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.349803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.349872] Lustre: Skipped 1 previous similar message [ 2554.359054] Lustre: Skipped 1 previous similar message [ 2559.469453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.483636] Lustre: Skipped 2 previous similar messages [ 2560.714613] Lustre: server umount lustre-MDT0000 complete [ 2564.579207] LustreError: 95020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.590551] LustreError: 95020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2564.913604] LustreError: 93973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834152 with bad export cookie 2824982360554104610 [ 2564.915214] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.924037] LustreError: 93973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2565.229796] Lustre: server umount lustre-MDT0001 complete [ 2578.398253] Lustre: server umount lustre-OST0000 complete [ 2591.026992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.542213] LustreError: 98292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.726053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2596.454692] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2596.839786] LustreError: 98293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.955888] LustreError: 98292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.281632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.047330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2610.958651] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2618.663294] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2624.587614] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2640.286030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.592841] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2640.599224] LustreError: 98279:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2640.617768] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2640.628712] LustreError: 100376:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2640.636493] LustreError: 100376:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2640.647539] LustreError: 100376:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2640.835387] Lustre: server umount lustre-OST0000 complete [ 2640.856191] LustreError: 100376:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2650.005718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.086719] LustreError: 100850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.257614] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2650.270150] LustreError: 98280:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2650.298800] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2650.304334] LustreError: 100843:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2650.307821] LustreError: 100843:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2650.312350] LustreError: 100843:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2650.389736] Lustre: server umount lustre-OST0000 complete [ 2650.396515] LustreError: 100843:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2661.090348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.380656] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2661.386476] Lustre: Found index 0 for lustre-OST0000, updating log [ 2661.395488] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2661.505205] Lustre: lustre-OST0000: new disk, initializing [ 2661.509814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2661.521704] Lustre: 101379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2661.850302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.851756] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2674.048603] Lustre: Failing over lustre-OST0000 [ 2674.148801] Lustre: server umount lustre-OST0000 complete [ 2680.673663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.681279] Lustre: Skipped 1 previous similar message [ 2680.688483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.921046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.927906] Lustre: Skipped 2 previous similar messages [ 2688.029336] Lustre: server umount lustre-MDT0000 complete [ 2691.045082] LustreError: 98293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.075390] LustreError: 98293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2691.955837] LustreError: 98273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834279 with bad export cookie 2824982360554107354 [ 2691.966121] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.982032] LustreError: 98273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2692.351803] Lustre: server umount lustre-MDT0001 complete [ 2703.961905] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:38:10 (1787834290) [ 2721.270786] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2732.193053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.689103] LustreError: 103736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.828380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.544549] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2746.582996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.106147] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2755.300397] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2762.070843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2762.778098] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2769.058308] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2771.943074] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2771.953325] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2771.964707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2777.061827] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2777.085276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2777.952478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.027937] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2778.086380] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2778.113518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2778.143442] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2778.144526] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2778.159870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2778.184651] Lustre: lustre-OST0001: new disk, initializing [ 2778.201051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2778.207668] Lustre: 106058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2778.587478] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.592245] Lustre: Skipped 2 previous similar messages [ 2784.874608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2784.894876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2785.033417] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2785.139499] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2792.546939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2799.970415] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2805.738905] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.747498] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2805.759801] Lustre: Skipped 2 previous similar messages [ 2805.776074] Lustre: Skipped 2 previous similar messages [ 2810.671942] Lustre: server umount lustre-MDT0001 complete [ 2810.860528] LustreError: 107042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.896939] LustreError: 107042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2820.558269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.647064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2820.652085] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2820.715140] Lustre: 103726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2822.375777] LustreError: 107634:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2822.382277] LustreError: 107634:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2822.391574] LustreError: 103728:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2822.499919] LustreError: 107643:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2822.508139] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2822.508139] [ 2825.755332] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2826.743417] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2834.579666] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:40:20 (1787834420) [ 2851.177257] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2868.699290] Lustre: 109673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2891.991680] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2895.119367] Lustre: 110809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2984.928440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2984.932126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.945962] Lustre: Skipped 1 previous similar message [ 2984.955545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.964526] Lustre: Skipped 1 previous similar message [ 2988.540381] Lustre: server umount lustre-MDT0000 complete [ 2990.561884] LustreError: 107042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.572086] LustreError: 107042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2992.000084] LustreError: 106776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834579 with bad export cookie 2824982360554108355 [ 2992.007978] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2992.008245] LustreError: 106776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2992.373650] Lustre: server umount lustre-MDT0001 complete [ 3006.994606] Lustre: server umount lustre-OST0000 complete [ 3021.103986] Lustre: server umount lustre-OST0001 complete [ 3027.067162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3037.414250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3045.230502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3055.397210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3079.002495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3090.065555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.081424] Lustre: lustre-MDT0000: reset Object Index mappings [ 3090.748024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.753413] Lustre: Skipped 1 previous similar message [ 3095.614855] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3104.170106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.196251] Lustre: lustre-MDT0001: reset Object Index mappings [ 3108.986635] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3111.742663] Lustre: 115908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3118.700278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.031131] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3129.874431] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3134.581894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3136.312992] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3136.317443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3136.335384] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3141.891797] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3148.987452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3152.657917] Lustre: 117786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3204.635603] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:46:31 (1787834791) [ 3232.737285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3232.754095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.762563] Lustre: Skipped 3 previous similar messages [ 3232.773739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.781989] Lustre: Skipped 4 previous similar messages [ 3235.757930] Lustre: server umount lustre-MDT0000 complete [ 3239.276716] LustreError: 117789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834827 with bad export cookie 2824982360554155430 [ 3239.305796] LustreError: 117789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3239.310236] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3244.015144] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.029384] Lustre: Skipped 4 previous similar messages [ 3245.905488] Lustre: server umount lustre-MDT0001 complete [ 3256.351612] Lustre: server umount lustre-OST0000 complete [ 3259.915559] LustreError: 118746:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x36:0x0].0x0 (ffff9b3c05911f00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3274.207155] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3274.453645] Lustre: server umount lustre-OST0001 complete [ 3280.414400] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3290.025872] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3327.510281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.786351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3327.861509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3327.991286] Lustre: lustre-MDT0000: new disk, initializing [ 3328.196835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.206347] Lustre: Skipped 3 previous similar messages [ 3328.236398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3332.735274] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3340.852830] Lustre: server umount lustre-MDT0000 complete [ 3350.716354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.993124] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3351.011387] Lustre: 122056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3351.025819] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3355.736931] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3360.486082] Lustre: server umount lustre-MDT0000 complete [ 3369.687352] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:49:16 (1787834956) [ 3371.201982] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3373.152102] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:49:19 (1787834959) [ 3393.768509] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3403.697903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.589221] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3416.707139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.848540] Lustre: 124107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3416.912364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3416.916610] Lustre: Skipped 1 previous similar message [ 3416.981501] Lustre: lustre-MDT0001: new disk, initializing [ 3417.110899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3417.127196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3421.604920] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3426.138666] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3433.353661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3433.662452] Lustre: lustre-OST0000: new disk, initializing [ 3433.668136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3433.679832] Lustre: 125687:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3435.353161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3435.370274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3435.517873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3439.815424] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3450.326991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.551191] Lustre: lustre-OST0001: new disk, initializing [ 3450.555197] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3450.566929] Lustre: 126558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3451.775730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3451.788237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3451.961682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3456.856349] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3466.359808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3470.733517] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3871.799165] Lustre: 132977:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3871.808087] Lustre: 132977:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3871.818830] Lustre: 132977:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3871.828941] Lustre: 132977:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3871.838278] Lustre: 132977:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3871.843773] Lustre: 132977:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4176.175639] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:02:42 (1787835762) [ 4195.305096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.306144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.314303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4195.326748] Lustre: Skipped 2 previous similar messages [ 4195.350281] Lustre: Skipped 9 previous similar messages [ 4199.632163] Lustre: server umount lustre-MDT0000 complete [ 4200.417552] LustreError: 124120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.459917] LustreError: 124120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4204.041834] LustreError: 124099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835791 with bad export cookie 2824982360554223547 [ 4204.053911] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.058547] LustreError: 124099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4205.541753] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.547221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4205.563424] Lustre: Skipped 2 previous similar messages [ 4205.591649] Lustre: Skipped 1 previous similar message [ 4210.639374] Lustre: server umount lustre-MDT0001 complete [ 4220.547788] Lustre: server umount lustre-OST0000 complete [ 4230.743612] Lustre: server umount lustre-OST0001 complete [ 4237.974342] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4247.339850] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4283.652990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.780320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4283.801743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4283.852715] Lustre: lustre-MDT0000: new disk, initializing [ 4283.924095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.927077] Lustre: Skipped 5 previous similar messages [ 4283.943454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4288.788598] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4301.898572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.077976] Lustre: 136051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4302.168084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4302.176678] Lustre: Skipped 1 previous similar message [ 4302.276861] Lustre: lustre-MDT0001: new disk, initializing [ 4302.446857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4302.466175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4307.640667] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4318.101683] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4323.968398] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4331.920558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4332.395181] Lustre: lustre-OST0000: new disk, initializing [ 4332.402279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4332.427160] Lustre: 138019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4332.550069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.564178] Lustre: Skipped 1 previous similar message [ 4333.841451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4333.849436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4333.955669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4339.422260] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4349.383110] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4359.322200] Lustre: lustre-MDD0000: changelog on [ 4362.370738] Lustre: lustre-MDD0001: changelog on [ 6455.337842] Lustre: lustre-MDD0001: changelog off [ 6457.038216] Lustre: 136057:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6457.811570] Lustre: lustre-MDD0000: changelog off [ 6463.997306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.008067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6464.018285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.018293] Lustre: Skipped 2 previous similar messages [ 6464.019317] Lustre: Skipped 1 previous similar message [ 6464.051920] LustreError: Skipped 1 previous similar message [ 6469.090427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6469.096961] Lustre: Skipped 2 previous similar messages [ 6474.215207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6474.228822] Lustre: Skipped 1 previous similar message [ 6478.304294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6478.481927] Lustre: server umount lustre-OST0000 complete [ 6483.426813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6483.442623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.467513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6484.449666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6488.632461] Lustre: server umount lustre-MDT0000 complete [ 6489.569608] LustreError: 137364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.591519] LustreError: 137364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6492.205551] LustreError: 136041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838080 with bad export cookie 2824982360555047083 [ 6492.214227] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6492.216156] LustreError: 136041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6492.572476] Lustre: server umount lustre-MDT0001 complete [ 6503.795457] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 6506.452488] Key type lgssc unregistered [ 6506.704150] LNet: 147185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6506.714730] LNetError: 147185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6506.729581] LNet: Removed LNI 192.168.203.144@tcp [ 6507.539308] Key type .llcrypt unregistered [ 6507.542486] Key type ._llcrypt unregistered [ 6529.643858] Key type ._llcrypt registered [ 6529.650817] Key type .llcrypt registered [ 6529.776438] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6546.350027] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6547.701711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6547.820068] alg: No test for adler32 (adler32-zlib) [ 6548.901631] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6549.197461] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 6550.871203] Key type lgssc registered [ 6552.059492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6588.227050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6588.252234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.473878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6589.514102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6589.598156] Lustre: lustre-MDT0000: new disk, initializing [ 6589.661971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.679561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6594.660841] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6607.550493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.713041] Lustre: 150387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6607.789462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6607.803250] Lustre: Skipped 1 previous similar message [ 6607.934256] Lustre: lustre-MDT0001: new disk, initializing [ 6608.080131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6608.117848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6608.127656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6612.660227] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6623.353238] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6629.263732] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6636.084885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.475552] Lustre: lustre-OST0000: new disk, initializing [ 6636.483624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6636.491423] Lustre: 152357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6636.600303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.955254] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6645.788078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6645.800183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6645.876709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6653.072765] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6659.172644] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module obdecho/obdecho [ 6661.140812] LustreError: 153360:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6661.148277] LustreError: 153360:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6662.755794] LustreError: 153457:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6676.455303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.460195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6676.483856] Lustre: Skipped 2 previous similar messages [ 6676.507663] Lustre: Skipped 2 previous similar messages [ 6681.568182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.571021] Lustre: Skipped 2 previous similar messages [ 6682.255932] Lustre: server umount lustre-MDT0000 complete [ 6685.694490] LustreError: 150380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838273 with bad export cookie 2121407315036324733 [ 6685.695156] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6685.700536] LustreError: 150380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6685.963703] Lustre: server umount lustre-MDT0001 complete [ 6700.711395] Lustre: server umount lustre-OST0000 complete [ 6723.951354] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6735.382224] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6778.212262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.525249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6778.567686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6778.750142] Lustre: lustre-MDT0000: new disk, initializing [ 6778.884291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6778.925450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6783.501462] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6796.926609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6797.210397] Lustre: 157176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6797.306341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6797.317326] Lustre: Skipped 1 previous similar message [ 6797.423296] Lustre: lustre-MDT0001: new disk, initializing [ 6797.573127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6797.649466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6797.670742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6802.964968] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6815.233693] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6822.348542] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6829.887189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.255594] Lustre: lustre-OST0000: new disk, initializing [ 6830.259624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6830.273406] Lustre: 159143:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6830.390095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6837.182761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6838.863696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6838.880404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6839.024064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6847.026361] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6852.242093] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6852.500936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6857.347688] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6857.662038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6864.865287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6864.867286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.876405] LustreError: Skipped 1 previous similar message [ 6864.880613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6864.891859] Lustre: Skipped 1 previous similar message [ 6864.906036] Lustre: Skipped 1 previous similar message [ 6866.138899] Lustre: server umount lustre-OST0000 complete [ 6874.593420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6874.614821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.643280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.224881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.233263] Lustre: Skipped 2 previous similar messages [ 6885.343120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.590364] Lustre: server umount lustre-MDT0000 complete [ 6889.688469] LustreError: 157168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838477 with bad export cookie 2121407315036325685 [ 6889.689926] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6889.704698] LustreError: 157168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6890.111332] Lustre: server umount lustre-MDT0001 complete [ 6893.996273] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:48:00 (1787838480) [ 6895.723985] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6897.976703] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:48:04 (1787838484) [ 6907.117046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.559197] LustreError: 161559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.571159] LustreError: 161559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6907.619368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6912.390903] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6912.997189] LustreError: 161560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.099316] LustreError: 161559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.209470] LustreError: 161560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.472825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.230342] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6936.589818] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6943.060762] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6950.827821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6951.415334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.430650] Lustre: Skipped 1 previous similar message [ 6958.263784] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6967.613695] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6974.433229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6974.440439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.451887] Lustre: Skipped 1 previous similar message [ 6974.458362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6974.467507] Lustre: Skipped 2 previous similar messages [ 6980.360835] Lustre: server umount lustre-MDT0000 complete [ 6983.137979] LustreError: 162550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6983.161328] LustreError: 162550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6984.144158] LustreError: 161541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838572 with bad export cookie 2121407315036326574 [ 6984.154203] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6984.158920] LustreError: 161541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6988.269225] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.287158] Lustre: Skipped 2 previous similar messages [ 6990.581436] Lustre: server umount lustre-MDT0001 complete [ 6997.924360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7012.432421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.840118] LustreError: 165405:0:(llog.c:1655:llog_backup()) MGC192.168.203.144@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7012.849433] Lustre: 165405:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7014.880571] LustreError: 165410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.925177] LustreError: 165410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7034.336373] LustreError: 165410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.355093] LustreError: 165410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7070.176122] LustreError: 165441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7070.211684] LustreError: 165441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 7073.759652] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7073.765566] Lustre: 165405:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7114.758350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7119.858515] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7119.922714] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7129.754635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.082921] LustreError: 148609:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000470c5c8b x1874684162475136/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7135.736800] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7135.962050] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7144.586734] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7146.499539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7154.275464] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7156.823254] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7185.496164] Lustre: server umount lustre-MDT0000 complete [ 7186.914116] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.916640] LustreError: 163393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7186.933755] Lustre: Skipped 1 previous similar message [ 7186.953594] LustreError: 163393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7189.112951] LustreError: 161540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838777 with bad export cookie 2121407315036333028 [ 7189.123559] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7189.124443] LustreError: 161540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7189.486125] Lustre: server umount lustre-MDT0001 complete [ 7204.276072] Lustre: server umount lustre-OST0000 complete [ 7228.702632] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7239.350202] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7281.519815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7281.957153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7281.984981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7282.090989] Lustre: lustre-MDT0000: new disk, initializing [ 7282.219080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7282.231986] Lustre: Skipped 1 previous similar message [ 7282.271256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7288.657827] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7301.507157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.730376] Lustre: 170510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7302.036453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7308.284761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7319.127453] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7324.239565] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7330.030577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7330.308282] Lustre: lustre-OST0000: new disk, initializing [ 7330.311261] Lustre: Skipped 1 previous similar message [ 7330.314686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7330.321210] Lustre: Skipped 2 previous similar messages [ 7330.328949] Lustre: 172481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7336.031251] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7339.623908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7339.634655] Lustre: Skipped 1 previous similar message [ 7339.643383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7339.758583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7345.141271] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7349.864421] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7350.061287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7354.463851] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7354.597651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7360.492376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7360.500638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7360.507597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7360.515235] Lustre: Skipped 2 previous similar messages [ 7360.564482] Lustre: Skipped 8 previous similar messages [ 7362.574285] Lustre: server umount lustre-OST0000 complete [ 7368.674861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7368.695178] LustreError: Skipped 1 previous similar message [ 7375.198946] Lustre: server umount lustre-MDT0000 complete [ 7375.851528] LustreError: 170523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.875672] LustreError: 170523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7379.584123] LustreError: 170504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838967 with bad export cookie 2121407315036349863 [ 7379.589859] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7379.593054] LustreError: 170504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7379.947974] Lustre: server umount lustre-MDT0001 complete [ 7388.613770] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:56:15 (1787838975) [ 7395.308500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7400.398529] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7408.374921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.479056] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7421.598596] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7427.375558] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7433.566773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.837910] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7447.506884] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7458.067387] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:57:24 (1787839044) [ 7464.945258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7464.959880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.975230] Lustre: Skipped 2 previous similar messages [ 7464.984196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7464.989619] Lustre: Skipped 3 previous similar messages [ 7469.451084] Lustre: server umount lustre-OST0000 complete [ 7480.546362] Lustre: server umount lustre-MDT0000 complete [ 7484.719559] LustreError: 174873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839072 with bad export cookie 2121407315036350745 [ 7484.726555] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7484.732882] LustreError: 174873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7485.100939] Lustre: server umount lustre-MDT0001 complete [ 7496.423111] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7499.413392] Key type lgssc unregistered [ 7499.684592] LNet: 178620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7499.699130] LNetError: 178620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7499.754971] LNet: Removed LNI 192.168.203.144@tcp [ 7500.776045] Key type .llcrypt unregistered [ 7500.782213] Key type ._llcrypt unregistered [ 7520.090498] Key type ._llcrypt registered [ 7520.092564] Key type .llcrypt registered [ 7520.211122] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7521.243354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7521.392529] alg: No test for adler32 (adler32-zlib) [ 7522.444414] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7522.686934] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7524.407171] Key type lgssc registered [ 7525.417066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7532.888867] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7532.925818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.447390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7553.568652] LustreError: 179866:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 7558.687471] LustreError: 179866:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7558.710329] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7558.713016] LustreError: MGC192.168.203.144@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7558.719255] LustreError: 179866:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7558.724349] LustreError: 179866:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7558.729064] LustreError: 179866:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7558.781232] Lustre: server umount lustre-OST0000 complete [ 7558.783375] LustreError: 179866:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7566.662461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7567.993669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7572.626280] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7577.570908] LustreError: 180367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7577.584917] LustreError: 180367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7581.379643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.778117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7585.997734] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7592.763459] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7597.928274] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7604.315366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7604.560328] LustreError: 182181:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7604.573149] Lustre: 182181:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.144@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7604.690820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.714424] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7618.144549] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7622.625220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7622.643949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.665618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7626.790245] Lustre: server umount lustre-MDT0000 complete [ 7628.259537] LustreError: 180362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.274618] LustreError: 180362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7630.182546] LustreError: 182208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839218 with bad export cookie 6992997838948647800 [ 7630.195437] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7630.454161] Lustre: server umount lustre-MDT0001 complete [ 7644.745571] Lustre: server umount lustre-OST0000 complete [ 7654.319188] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7664.102107] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7701.743719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7702.042970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7702.070474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7702.198700] Lustre: lustre-MDT0000: new disk, initializing [ 7702.294153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7702.313430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7707.408324] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7717.874299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.010138] Lustre: 185719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7718.080594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7718.083917] Lustre: Skipped 1 previous similar message [ 7718.186920] Lustre: lustre-MDT0001: new disk, initializing [ 7718.316662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7718.359244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7718.378928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7723.235506] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7733.554883] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7740.553309] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7747.044836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7747.342561] Lustre: lustre-OST0000: new disk, initializing [ 7747.347291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7747.353201] Lustre: 187689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7747.406756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.304579] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7755.332126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7755.343102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7755.434656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7762.786318] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7767.378971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7767.571289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7771.653916] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7771.881509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7776.224805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7776.224929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.230906] LustreError: Skipped 1 previous similar message [ 7776.237384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.249327] Lustre: Skipped 3 previous similar messages [ 7776.262490] Lustre: Skipped 4 previous similar messages [ 7780.092993] Lustre: server umount lustre-OST0000 complete [ 7784.929465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7784.935062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7784.944602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7791.586182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7791.595898] Lustre: Skipped 3 previous similar messages [ 7796.706917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.723102] Lustre: Skipped 1 previous similar message [ 7798.239134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7798.569122] Lustre: server umount lustre-MDT0000 complete [ 7801.829992] LustreError: 187150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.853957] LustreError: 187150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7802.211976] LustreError: 185714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839390 with bad export cookie 6992997838948648521 [ 7802.213037] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7802.220091] LustreError: 185714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7802.697824] Lustre: server umount lustre-MDT0001 complete [ 7811.397484] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:03:18 (1787839398) [ 7828.165509] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7830.636635] Key type lgssc unregistered [ 7830.886634] LNet: 190638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7830.903994] LNetError: 190638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7830.924264] LNet: Removed LNI 192.168.203.144@tcp [ 7831.668223] Key type .llcrypt unregistered [ 7831.670585] Key type ._llcrypt unregistered [ 7848.468791] Key type ._llcrypt registered [ 7848.472416] Key type .llcrypt registered [ 7848.569610] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7849.555804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7849.579231] alg: No test for adler32 (adler32-zlib) [ 7850.704784] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7851.338052] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7853.111193] Key type lgssc registered [ 7854.537179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7865.015373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7865.046997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.713765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.775399] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7876.146540] LustreError: 191856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7876.162337] LustreError: 191856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7881.189857] LustreError: 191857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.605547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.974811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7887.592934] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7897.555273] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7903.891179] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7911.058303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7911.243108] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7911.247091] LustreError: 193722:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7911.255956] Lustre: 193722:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.144@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7911.371923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.134774] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7922.968974] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7927.781487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7927.785653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7927.790803] Lustre: Skipped 2 previous similar messages [ 7927.801848] Lustre: Skipped 2 previous similar messages [ 7928.804867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7928.819049] Lustre: Skipped 2 previous similar messages [ 7931.036208] Lustre: server umount lustre-MDT0000 complete [ 7933.926696] LustreError: 191851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.947252] LustreError: 191851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7934.233779] LustreError: 191835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839522 with bad export cookie 17657052897210344483 [ 7934.241184] LustreError: 191835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7934.241479] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.007289] LustreError: 192587:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001bea4273 x1874685527618048/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7935.154787] Lustre: server umount lustre-MDT0001 complete [ 7948.809547] Lustre: server umount lustre-OST0000 complete [ 7958.524953] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7969.070462] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8007.710374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.929803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8007.955185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8008.066338] Lustre: lustre-MDT0000: new disk, initializing [ 8008.206785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.247273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8012.491735] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8022.917886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8023.021751] Lustre: 197242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8023.079185] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8023.088151] Lustre: Skipped 1 previous similar message [ 8023.091871] LustreError: 198025:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8023.103426] LustreError: 198025:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8023.114194] Lustre: 198025:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8023.149278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8023.158128] Lustre: Skipped 1 previous similar message [ 8023.237267] Lustre: lustre-MDT0001: new disk, initializing [ 8023.313331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8023.342459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8023.366340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8027.949198] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8037.738814] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8044.063154] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8051.054802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.252895] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8051.255755] Lustre: Skipped 1 previous similar message [ 8051.258575] LustreError: 199190:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8051.264697] LustreError: 199190:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8051.269573] Lustre: 199190:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8051.280017] Lustre: 199190:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8051.297595] Lustre: lustre-OST0000: new disk, initializing [ 8051.301208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8051.306612] Lustre: 199213:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8051.383840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.406812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8057.410267] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8057.414654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8057.526117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8066.700171] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8071.863430] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8072.086715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8076.796652] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8077.052373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8083.427142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8083.432799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8083.452119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8083.456830] Lustre: Skipped 1 previous similar message [ 8088.558934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8088.570430] Lustre: Skipped 1 previous similar message [ 8093.151415] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8093.341521] Lustre: server umount lustre-OST0000 complete [ 8098.784277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.793646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.815342] Lustre: Skipped 2 previous similar messages [ 8103.349692] Lustre: server umount lustre-MDT0000 complete [ 8103.905207] LustreError: 197251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.927339] LustreError: 197251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8106.819278] LustreError: 197234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839694 with bad export cookie 17657052897210345204 [ 8106.833527] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8107.061222] Lustre: server umount lustre-MDT0001 complete [ 8115.053483] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:08:21 (1787839701) [ 8132.768914] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8143.822706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.124269] LustreError: 202339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.152860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8149.787454] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8154.601300] LustreError: 202340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8154.632937] LustreError: 202340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8158.510384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.846716] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8166.191670] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8174.266839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8181.718988] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8190.941724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8191.128762] Lustre: lustre-OST0001: new disk, initializing [ 8191.132762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8191.141846] Lustre: 204650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8191.247477] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8191.258171] Lustre: Skipped 2 previous similar messages [ 8197.674172] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8200.301047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8200.310964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8200.385055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8208.149258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8218.223839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8220.201962] Lustre: DEBUG MARKER: ADD OST3 [ 8225.142258] loop: module loaded [ 8225.245925] loop0: detected capacity change from 0 to 204800000 [ 8225.483307] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8225.499675] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8225.518121] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8231.613772] loop0: detected capacity change from 0 to 204800000 [ 8232.619468] Lustre: DEBUG MARKER: STOP OST3 [ 8237.169532] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8237.345476] Lustre: lustre-OST0003: new disk, initializing [ 8237.348181] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8237.358169] Lustre: 206640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8239.376629] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8239.443922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8239.451332] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8239.485341] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8241.112639] Lustre: Failing over lustre-OST0003 [ 8241.327893] Lustre: server umount lustre-OST0003 complete [ 8241.439082] LustreError: 206856:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8243.656320] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8244.303815] LustreError: 203824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.704450] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8244.713645] LustreError: Skipped 1 previous similar message [ 8244.720347] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.026619] Lustre: DEBUG MARKER: START OST3 again [ 8258.631281] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8258.911404] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8258.917699] Lustre: Skipped 1 previous similar message [ 8258.931627] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8259.645386] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8260.413638] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8260.413954] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8264.455754] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8284.500154] Lustre: Failing over lustre-OST0003 [ 8284.561600] Lustre: server umount lustre-OST0003 complete [ 8284.647645] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8284.651458] LustreError: 203826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.651724] Lustre: Skipped 2 previous similar messages [ 8284.676741] LustreError: 203826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8302.560388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8302.574079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.597419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8302.613731] Lustre: Skipped 1 previous similar message [ 8308.766516] Lustre: server umount lustre-MDT0000 complete [ 8313.525770] LustreError: 202319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839901 with bad export cookie 17657052897210346086 [ 8313.543296] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8313.548808] LustreError: 202319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8320.176098] Lustre: server umount lustre-MDT0001 complete [ 8324.880097] Lustre: server umount lustre-OST0000 complete [ 8328.464900] Lustre: server umount lustre-OST0001 complete [ 8335.762459] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8343.453808] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8376.966243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8377.328741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8377.374847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8377.516560] Lustre: lustre-MDT0000: new disk, initializing [ 8377.630602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8381.860859] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8391.906696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8392.045938] Lustre: 212110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8392.316130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8392.327920] Lustre: Skipped 1 previous similar message [ 8392.373173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8397.158921] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8406.811201] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8412.482912] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8419.573227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.840136] Lustre: lustre-OST0000: new disk, initializing [ 8419.848379] Lustre: Skipped 1 previous similar message [ 8419.853236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8419.865321] Lustre: Skipped 2 previous similar messages [ 8419.876277] Lustre: 214077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8421.761544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8421.769425] Lustre: Skipped 1 previous similar message [ 8421.787117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8421.877645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8426.470695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8436.590489] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8441.917081] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8442.161213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8447.227219] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8447.512935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8452.583245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8452.586298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.587028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8452.587034] Lustre: Skipped 8 previous similar messages [ 8452.644151] Lustre: Skipped 6 previous similar messages [ 8455.737218] Lustre: server umount lustre-OST0000 complete [ 8465.786717] Lustre: server umount lustre-MDT0000 complete [ 8467.938082] LustreError: 213602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8467.973952] LustreError: 213602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8469.755946] LustreError: 212101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840057 with bad export cookie 17657052897210348382 [ 8469.756997] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8469.771224] LustreError: 212101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8470.218946] Lustre: server umount lustre-MDT0001 complete [ 8473.782895] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:14:20 (1787840060) [ 8489.026392] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8499.936279] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8535.369796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.563917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8535.595661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8535.686502] Lustre: lustre-MDT0000: new disk, initializing [ 8535.801198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8541.349649] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8553.939308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.023499] Lustre: 218689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8554.340170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8559.265269] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8568.147219] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8573.982993] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8580.894522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.127905] Lustre: 220656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8583.053114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8587.085148] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8595.874204] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8600.311566] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8600.516579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8604.815650] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8605.101989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8608.738381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8608.751699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8608.752700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8608.752706] Lustre: Skipped 3 previous similar messages [ 8608.761213] Lustre: Skipped 3 previous similar messages [ 8608.791970] LustreError: Skipped 1 previous similar message [ 8613.059720] Lustre: server umount lustre-OST0000 complete [ 8623.772846] Lustre: server umount lustre-MDT0000 complete [ 8624.099137] LustreError: 219542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8624.145692] LustreError: 219542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8627.525078] LustreError: 218679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840215 with bad export cookie 17657052897210349264 [ 8627.538431] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8627.564288] LustreError: 218679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8627.853650] Lustre: server umount lustre-MDT0001 complete [ 8637.881518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.356966] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8654.229639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.744697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8654.760279] Lustre: Skipped 5 previous similar messages [ 8658.638467] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8666.121886] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8672.135918] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8678.974541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8686.106110] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8694.824051] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8706.017981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8706.024297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.032986] Lustre: Skipped 2 previous similar messages [ 8706.039018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8706.043440] Lustre: Skipped 3 previous similar messages [ 8710.818938] Lustre: server umount lustre-OST0000 complete [ 8721.454422] Lustre: server umount lustre-MDT0000 complete [ 8725.289206] LustreError: 222851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840313 with bad export cookie 17657052897210350146 [ 8725.293373] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8725.296494] LustreError: 222851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8725.607955] Lustre: server umount lustre-MDT0001 complete [ 8835.583192] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8838.398576] Key type lgssc unregistered [ 8838.819701] LNet: 226541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8838.838670] LNetError: 226541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8838.873913] LNet: Removed LNI 192.168.203.144@tcp [ 8839.950127] Key type .llcrypt unregistered [ 8839.953472] Key type ._llcrypt unregistered [ 8867.463280] Key type ._llcrypt registered [ 8867.470262] Key type .llcrypt registered [ 8867.593192] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8888.071576] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8888.971255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8889.013940] alg: No test for adler32 (adler32-zlib) [ 8890.264927] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 8890.705352] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8892.431155] Key type lgssc registered [ 8893.836946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8928.261911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8928.309138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.536170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8929.559039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8929.618414] Lustre: lustre-MDT0000: new disk, initializing [ 8929.672234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8929.703088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8933.633223] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8943.160404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.334884] Lustre: 229892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8943.404304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8943.408574] Lustre: Skipped 1 previous similar message [ 8943.525995] Lustre: lustre-MDT0001: new disk, initializing [ 8943.702289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8943.746630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8943.758855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8948.040184] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8957.316460] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8963.083039] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8969.533564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8969.769314] Lustre: lustre-OST0000: new disk, initializing [ 8969.772801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8969.778100] Lustre: 231862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8969.841747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8975.804898] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8975.930877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8975.949974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8976.124659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8984.325944] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8988.398696] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8988.558149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8992.639229] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8992.776074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8996.832944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8996.846864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.857025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8996.860086] Lustre: Skipped 1 previous similar message [ 8996.879241] Lustre: Skipped 1 previous similar message [ 9001.215626] Lustre: server umount lustre-OST0000 complete [ 9007.073168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.080486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9007.094983] Lustre: Skipped 1 previous similar message [ 9011.594328] Lustre: server umount lustre-MDT0000 complete [ 9015.842920] LustreError: 229886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840603 with bad export cookie 14002752845094350481 [ 9015.866373] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9015.878371] LustreError: 229886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9016.317357] Lustre: server umount lustre-MDT0001 complete [ 9036.742682] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9046.846662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.244707] LustreError: 234795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.268780] LustreError: 234795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9047.327116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.968261] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9052.641899] LustreError: 234796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9057.760178] LustreError: 234795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.452020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9060.777966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9064.964047] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9068.757465] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9077.219929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9077.696185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.245350] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9094.599150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9094.739891] Lustre: lustre-OST0001: new disk, initializing [ 9094.743302] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9094.747956] Lustre: 237103:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9094.835742] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9104.039587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9104.048161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9104.152361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9104.423902] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9114.566941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9126.461479] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9138.473426] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:25:25 (1787840725) [ 9142.755393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9142.767179] LustreError: Skipped 1 previous similar message [ 9142.771610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.788129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9142.799523] Lustre: Skipped 1 previous similar message [ 9145.318504] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9145.322483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9145.325352] Lustre: Skipped 1 previous similar message [ 9145.328579] Lustre: Skipped 1 previous similar message [ 9146.916476] Lustre: server umount lustre-MDT0000 complete [ 9150.444137] LustreError: 234796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9150.466491] LustreError: 234796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9150.469583] LustreError: 237871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840738 with bad export cookie 14002752845094351363 [ 9150.481951] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9150.487371] LustreError: 237871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9150.828706] Lustre: server umount lustre-MDT0001 complete [ 9164.922789] Lustre: server umount lustre-OST0000 complete [ 9179.671341] Lustre: server umount lustre-OST0001 complete [ 9185.971191] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9194.923409] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9235.996113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9236.223326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9236.245357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9236.316152] Lustre: lustre-MDT0000: new disk, initializing [ 9236.406313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9236.463919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9240.917614] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9252.629487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9252.848460] Lustre: 241052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9252.961543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9252.967727] Lustre: Skipped 1 previous similar message [ 9253.103653] Lustre: lustre-MDT0001: new disk, initializing [ 9253.282864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9253.293832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9259.529207] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9271.528299] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9278.835087] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9287.558207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9287.805214] Lustre: lustre-OST0000: new disk, initializing [ 9287.810842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9287.823086] Lustre: 243022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9289.163354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9289.191086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9289.338961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9293.972673] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9304.992898] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9310.932409] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9311.204635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9315.576628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9315.733205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9319.909203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9319.910098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.919159] LustreError: Skipped 1 previous similar message [ 9319.921134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9319.921141] Lustre: Skipped 2 previous similar messages [ 9319.986018] Lustre: Skipped 2 previous similar messages [ 9323.463670] Lustre: server umount lustre-OST0000 complete [ 9330.145746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9330.151760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9330.155070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9330.156836] Lustre: Skipped 1 previous similar message [ 9330.162137] Lustre: Skipped 1 previous similar message [ 9333.468893] Lustre: server umount lustre-MDT0000 complete [ 9335.264935] LustreError: 243029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.316050] LustreError: 243029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9337.589215] LustreError: 241045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840925 with bad export cookie 14002752845094353498 [ 9337.593833] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9337.601335] LustreError: 241045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9337.867620] Lustre: server umount lustre-MDT0001 complete [ 9348.605829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.355355] LustreError: 245240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.528956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9349.537146] Lustre: Skipped 2 previous similar messages [ 9354.476872] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9364.752351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9370.637443] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9378.903752] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9384.548559] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9391.945480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9398.966437] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9406.697937] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9554.401212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9554.418645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.435930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9554.440185] Lustre: Skipped 1 previous similar message [ 9556.715512] Lustre: server umount lustre-OST0000 complete [ 9565.152557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9565.158478] LustreError: Skipped 1 previous similar message [ 9568.485650] Lustre: server umount lustre-MDT0000 complete [ 9570.784729] LustreError: 246224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9570.816338] LustreError: 246224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9573.802866] LustreError: 245221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841161 with bad export cookie 14002752845094354380 [ 9573.806680] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9573.815379] LustreError: 245221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9574.138431] Lustre: server umount lustre-MDT0001 complete [ 9674.924428] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9677.693229] Key type lgssc unregistered [ 9677.999528] LNet: 248850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9678.004755] LNetError: 248850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9678.023772] LNet: Removed LNI 192.168.203.144@tcp [ 9679.134177] Key type .llcrypt unregistered [ 9679.138119] Key type ._llcrypt unregistered [ 9705.391221] Key type ._llcrypt registered [ 9705.397208] Key type .llcrypt registered [ 9705.583894] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9724.292454] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9725.523525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9725.651253] alg: No test for adler32 (adler32-zlib) [ 9726.710281] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 9726.992475] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9728.695252] Key type lgssc registered [ 9730.173122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9785.085955] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9796.231076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9796.268438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.432278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9797.484375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9797.594364] Lustre: lustre-MDT0000: new disk, initializing [ 9797.779752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.914838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9803.580844] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9817.008618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9817.119925] Lustre: 252922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9817.191476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9817.198833] Lustre: Skipped 1 previous similar message [ 9817.272207] Lustre: lustre-MDT0001: new disk, initializing [ 9817.371989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9817.420285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9817.431742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9822.091152] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9827.038685] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9832.735186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9832.989093] Lustre: lustre-OST0000: new disk, initializing [ 9832.995517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9833.002171] Lustre: 254555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9833.101659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9839.085221] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9840.185186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9840.191610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9840.273075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9849.451199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9849.636335] Lustre: lustre-OST0001: new disk, initializing [ 9849.641049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9849.648456] Lustre: 255425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9849.811309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9856.451126] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9857.579714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9857.587722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9857.675966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9866.816743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9876.969297] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9888.737537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9888.753610] Lustre: Skipped 1 previous similar message [ 9888.753798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9888.762757] Lustre: Skipped 2 previous similar messages [ 9893.861213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9893.878713] Lustre: Skipped 3 previous similar messages [ 9894.820091] Lustre: server umount lustre-MDT0000 complete [ 9898.538918] LustreError: 254554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841486 with bad export cookie 5990171764530819713 [ 9898.549499] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.561754] LustreError: 254554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9898.916730] Lustre: server umount lustre-MDT0001 complete [ 9913.037475] Lustre: server umount lustre-OST0000 complete [ 9914.273711] Lustre: 250430:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841486/real 1787841486] req@ffff9b3c2f62b0c0 x1874687493746176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9914.301384] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9914.316645] Lustre: Skipped 1 previous similar message [ 9917.195073] Lustre: server umount lustre-OST0001 complete [ 9931.316882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9941.361933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9949.654922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9958.154111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9974.772339] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9983.843320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.878250] Lustre: lustre-MDT0000: reset Object Index mappings [ 9984.270097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9988.293488] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9989.601311] LustreError: 260436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.623273] LustreError: 260436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9994.733080] LustreError: 260435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9997.025610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9997.044220] Lustre: lustre-MDT0001: reset Object Index mappings [ 9997.349951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10001.604594] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10004.738158] Lustre: 261577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10010.961496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10017.291241] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10022.631453] LustreError: 261932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.450916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10025.710898] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10025.718498] Lustre: Skipped 1 previous similar message [10029.809574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10029.840993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10031.881129] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10039.478319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10042.836519] Lustre: 263448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10048.481896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10048.488124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10048.498057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10050.116166] Lustre: server umount lustre-MDT0000 complete [10050.536380] LustreError: 260435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10050.542498] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10050.592949] Lustre: Skipped 2 previous similar messages [10054.141800] LustreError: 260416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841642 with bad export cookie 5990171764530826503 [10054.147996] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10054.161610] LustreError: 260416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10054.645071] Lustre: server umount lustre-MDT0001 complete [10069.125704] Lustre: server umount lustre-OST0000 complete [10072.101481] Lustre: 250431:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841643/real 1787841643] req@ffff9b3c3169e1c0 x1874687493792768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10072.121649] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.588374] Lustre: server umount lustre-OST0001 complete [10089.462434] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:41:16 (1787841676) [10107.908807] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [10119.476733] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10158.495921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.776045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10158.811491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10158.879299] Lustre: lustre-MDT0000: new disk, initializing [10158.981780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10159.016892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10164.620997] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10177.200161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.339956] Lustre: 267701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10177.392796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10177.396693] Lustre: Skipped 1 previous similar message [10177.456176] Lustre: lustre-MDT0001: new disk, initializing [10177.712169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10177.721532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10183.066475] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10194.031397] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10200.870743] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10209.724959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10210.071627] Lustre: lustre-OST0000: new disk, initializing [10210.079894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10210.094083] Lustre: 269673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10211.491161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10211.522337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10211.662475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10217.657186] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10228.512453] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10233.634358] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10233.869436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10238.561590] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10238.773193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10242.533251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.563175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10242.566610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10242.590830] Lustre: Skipped 1 previous similar message [10247.655680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10247.662539] Lustre: Skipped 1 previous similar message [10255.327116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10255.532321] Lustre: server umount lustre-OST0000 complete [10263.018149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10263.038541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10263.039876] Lustre: Skipped 2 previous similar messages [10263.050905] Lustre: Skipped 3 previous similar messages [10266.030748] Lustre: server umount lustre-MDT0000 complete [10268.131503] LustreError: 267715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10268.151375] LustreError: 267715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10270.300992] LustreError: 267695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841858 with bad export cookie 5990171764530828596 [10270.302900] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10270.328957] LustreError: 267695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10270.721975] Lustre: server umount lustre-MDT0001 complete [10292.698906] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10306.944368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.431533] LustreError: 272610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.465743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10307.468097] Lustre: Skipped 2 previous similar messages [10312.478061] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10321.076188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.829719] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10330.438226] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10338.885023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10347.564038] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10358.834981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10359.054332] Lustre: lustre-OST0001: new disk, initializing [10359.057516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10359.062118] Lustre: 274921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10360.735943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10360.745767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10360.790333] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10365.257778] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10375.495990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10379.419784] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10385.589147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10385.620731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10393.058265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10393.077696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.107651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.912732] Lustre: server umount lustre-MDT0000 complete [10401.767336] LustreError: 272605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.778227] LustreError: 272605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10402.781186] LustreError: 272592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841990 with bad export cookie 5990171764530829471 [10402.791205] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10402.802377] LustreError: 272592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10403.195510] Lustre: server umount lustre-MDT0001 complete [10417.636590] Lustre: server umount lustre-OST0000 complete [10432.092836] Lustre: server umount lustre-OST0001 complete [10452.558969] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10463.929122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.559194] LustreError: 277978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10464.650427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10464.655956] Lustre: Skipped 3 previous similar messages [10469.503345] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10480.438926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.345328] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10490.109773] Lustre: 279120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10499.431516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.419650] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10509.247903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10516.341789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10522.100113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10522.946390] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10531.355207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10535.903591] Lustre: 280992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10541.399408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10541.419318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10552.050949] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:48:58 (1787842138) [10567.652918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10567.664166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.679771] Lustre: Skipped 3 previous similar messages [10567.694091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10567.702071] Lustre: Skipped 4 previous similar messages [10570.127246] Lustre: server umount lustre-MDT0000 complete [10573.283140] LustreError: 277978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10573.317614] LustreError: 277978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10573.836398] LustreError: 277960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842161 with bad export cookie 5990171764530836254 [10573.858414] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.128079] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10588.447204] Lustre: server umount lustre-MDT0001 complete [10599.359390] Lustre: server umount lustre-OST0000 complete [10604.142364] Lustre: server umount lustre-OST0001 complete [10612.321452] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [10624.502440] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10671.624798] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10681.677906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.022652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10682.049394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10682.166889] Lustre: lustre-MDT0000: new disk, initializing [10682.319959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10687.906525] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10698.736490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.909893] Lustre: 285236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10699.279508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10703.546513] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10715.043862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.294623] Lustre: 286870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10717.529863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10721.816381] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10732.323656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10732.577704] Lustre: 287742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10732.722799] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10732.730373] Lustre: Skipped 6 previous similar messages [10734.301975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10734.323539] Lustre: Skipped 1 previous similar message [10734.467184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10740.004971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10749.791430] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10753.811533] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10753.815505] Lustre: Skipped 1 previous similar message [10769.041149] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:52:35 (1787842355) [10770.919248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10770.930968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.945339] Lustre: Skipped 5 previous similar messages [10770.955573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.958535] Lustre: Skipped 8 previous similar messages [10773.261200] Lustre: server umount lustre-MDT0000 complete [10775.536391] LustreError: 285249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.557182] LustreError: 285249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10776.978044] LustreError: 287741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842364 with bad export cookie 5990171764530843681 [10776.987025] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10776.993042] LustreError: 287741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10777.356209] Lustre: server umount lustre-MDT0001 complete [10790.347972] Lustre: server umount lustre-OST0000 complete [10803.983767] Lustre: server umount lustre-OST0001 complete [10812.702480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.307299] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10827.306270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10831.467262] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10838.690604] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10844.242082] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10850.573787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10857.403779] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10865.720729] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10882.439144] loop1: detected capacity change from 0 to 204800000 [10882.787041] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10882.802610] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10882.823596] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10888.274995] loop1: detected capacity change from 0 to 204800000 [10893.797480] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10893.904739] LustreError: 289979:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs32278: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10893.921087] LustreError: 289979:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs32278 [10893.923904] LustreError: 289979:0:(mgs_handler.c:517:mgs_target_reg()) fs32278-MDT0001: Error checking target index: rc = -13 [10893.931374] LustreError: fs32278-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10893.938580] LustreError: 293709:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10893.943318] LustreError: 293709:0:(tgt_mount.c:2129:server_put_super()) no obd fs32278-MDT0001 [10893.946851] LustreError: 293709:0:(tgt_mount.c:116:server_deregister_mount()) fs32278-MDT0001 not registered [10893.975818] Lustre: server umount fs32278-MDT0001 complete [10893.980580] LustreError: 293709:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10904.397943] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.759401] Lustre: Setting parameter fs32278-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32278-MDT0001 [10904.896726] Lustre: srv-fs32278-MDT0001: No data found on store. Initialize space. [10904.917512] Lustre: Skipped 4 previous similar messages [10905.037602] Lustre: fs32278-MDT0001: new disk, initializing [10905.042500] Lustre: Skipped 3 previous similar messages [10905.241683] Lustre: srv-fs32278-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10905.254123] Lustre: 294176:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32278-MDT0001-mdtlov: can't create new llog: rc = -115 [10905.271485] LustreError: 294176:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32278-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10911.798325] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10920.395422] Lustre: server umount fs32278-MDT0001 complete [10935.342354] loop1: detected capacity change from 0 to 204800000 [10940.394167] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10941.205535] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.609894] Lustre: 295744:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32278-MDT0001-mdtlov: can't create new llog: rc = -115 [10941.623202] LustreError: 295744:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32278-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10945.951651] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10950.410956] Lustre: server umount fs32278-MDT0001 complete [10960.864794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10960.875522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10960.882926] LustreError: Skipped 1 previous similar message [10960.898570] Lustre: Skipped 2 previous similar messages [10966.918662] Lustre: server umount lustre-MDT0000 complete [10976.771050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.886762] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10981.408362] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10982.391026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10989.342445] loop1: detected capacity change from 0 to 204800000 [10995.109140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.460365] Lustre: 298187:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32278-MDT0001-mdtlov: can't create new llog: rc = -115 [10995.474304] LustreError: 298187:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32278-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10999.747370] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11004.039944] Lustre: server umount fs32278-MDT0001 complete [11027.428210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11027.438281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11027.451209] Lustre: Skipped 6 previous similar messages [11031.422226] Lustre: server umount lustre-OST0000 complete [11041.420823] Lustre: server umount lustre-MDT0000 complete [11043.809275] LustreError: 289990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11043.826179] LustreError: 289990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 110 previous similar messages [11045.519362] LustreError: 289970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842633 with bad export cookie 5990171764530846362 [11045.530302] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11045.895785] Lustre: server umount lustre-MDT0001 complete [11057.717959] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11060.808795] Key type lgssc unregistered [11061.163708] LNet: 300537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11061.171691] LNetError: 300537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11061.191397] LNet: Removed LNI 192.168.203.144@tcp [11062.181488] Key type .llcrypt unregistered [11062.183920] Key type ._llcrypt unregistered [11082.522618] Key type ._llcrypt registered [11082.525746] Key type .llcrypt registered [11082.706618] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11084.561632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11084.607677] alg: No test for adler32 (adler32-zlib) [11086.166559] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11086.464921] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11088.200332] Key type lgssc registered [11089.430951] Lustre: Echo OBD driver; http://www.lustre.org/ [11098.668447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11098.716102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11100.277272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11104.446387] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11110.880763] LustreError: 301811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11110.903161] LustreError: 301811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11113.228057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11113.738919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11118.383647] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11126.478291] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11132.624035] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11139.693735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11140.220635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11145.321101] LustreError: 303640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11146.408646] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11150.822042] LustreError: 303641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.231263] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11155.903473] LustreError: 303639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.157207] LustreError: 303639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.183283] LustreError: 303639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11169.260185] LustreError: 303641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.286189] LustreError: 303641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11173.518955] loop1: detected capacity change from 0 to 204800000 [11173.684148] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11173.701133] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11173.717283] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11178.332793] loop1: detected capacity change from 0 to 204800000 [11183.017985] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11183.178907] Lustre: fs23380-OST0000: new disk, initializing [11183.182871] Lustre: srv-fs23380-OST0000: No data found on store. Initialize space. [11183.190702] Lustre: 305549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11183.259677] Lustre: fs23380-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11186.743607] LustreError: 303945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11186.771317] LustreError: 303945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11189.092047] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11197.110773] Lustre: server umount fs23380-OST0000 complete [11209.076399] loop1: detected capacity change from 0 to 204800000 [11214.760252] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11214.986205] Lustre: fs23380-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11220.459440] LustreError: 303640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.481313] LustreError: 303640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11221.500311] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11225.579326] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11229.492865] loop2: detected capacity change from 0 to 204800000 [11229.692754] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11229.710278] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11229.732270] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11234.394363] loop2: detected capacity change from 0 to 204800000 [11238.986307] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11239.069660] LustreError: fs23380-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11239.079605] LustreError: fs23380-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11239.088774] LustreError: 308589:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11239.095799] LustreError: 308589:0:(tgt_mount.c:2129:server_put_super()) no obd fs23380-OST0001 [11239.100866] LustreError: 308589:0:(tgt_mount.c:116:server_deregister_mount()) fs23380-OST0001 not registered [11239.130750] Lustre: server umount fs23380-OST0001 complete [11239.133945] LustreError: 308589:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11240.937996] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11249.979642] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11250.088197] Lustre: fs23380-OST0001: new disk, initializing [11250.093594] Lustre: srv-fs23380-OST0001: No data found on store. Initialize space. [11250.100475] Lustre: 309142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11250.205348] Lustre: fs23380-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11259.922727] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11269.911119] Lustre: server umount fs23380-OST0000 complete [11278.748691] Lustre: server umount fs23380-OST0001 complete [11287.011639] LustreError: 304456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11287.037160] LustreError: 304456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11289.748878] loop1: detected capacity change from 0 to 204800000 [11294.523814] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11294.807931] Lustre: fs23380-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11300.525433] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11306.733397] loop2: detected capacity change from 0 to 204800000 [11312.080863] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11318.388566] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11327.089700] Lustre: server umount fs23380-OST0000 complete [11335.669897] Lustre: server umount fs23380-OST0001 complete [11348.448747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11348.458781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11348.470470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11348.471359] Lustre: Skipped 1 previous similar message [11353.574145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11353.584084] Lustre: Skipped 2 previous similar messages [11354.904560] Lustre: server umount lustre-OST0000 complete [11359.712577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11359.724353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11359.744435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11363.808506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11363.811456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11365.391424] Lustre: server umount lustre-MDT0000 complete [11368.609630] LustreError: 301791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842956 with bad export cookie 16230927109339980967 [11368.615131] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11368.628668] LustreError: 301791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11368.990495] Lustre: server umount lustre-MDT0001 complete [11380.237579] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11383.641302] Key type lgssc unregistered [11383.944928] LNet: 315076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11383.952466] LNetError: 315076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11383.976210] LNet: Removed LNI 192.168.203.144@tcp [11384.956888] Key type .llcrypt unregistered [11384.967086] Key type ._llcrypt unregistered [11419.086606] Key type ._llcrypt registered [11419.090695] Key type .llcrypt registered [11419.209979] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11431.763380] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11432.801562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11432.822726] alg: No test for adler32 (adler32-zlib) [11433.855512] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11434.089084] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11435.735919] Key type lgssc registered [11436.712036] Lustre: Echo OBD driver; http://www.lustre.org/ [11468.996656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11469.036315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11470.271697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11470.318691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11470.391250] Lustre: lustre-MDT0000: new disk, initializing [11470.465627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11470.482767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11474.151205] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11483.587174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.669033] Lustre: 318914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11483.725682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11483.732873] Lustre: Skipped 1 previous similar message [11483.822882] Lustre: lustre-MDT0001: new disk, initializing [11483.906848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11483.953667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11483.969133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11488.044338] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11497.201372] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11502.216520] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11508.744501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11509.100560] Lustre: lustre-OST0000: new disk, initializing [11509.109526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11509.120492] Lustre: 320884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11509.202455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11515.292459] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11517.500744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11517.509610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11517.587744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11524.836289] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11527.099473] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b [11528.079756] Lustre: Unmounted lustre-client [11537.384492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.408381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11542.519057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11542.534338] Lustre: Skipped 2 previous similar messages [11543.484367] Lustre: server umount lustre-OST0000 complete [11547.616940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11547.640143] Lustre: Skipped 1 previous similar message [11547.646655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11552.738505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11552.752377] Lustre: Skipped 2 previous similar messages [11553.812748] Lustre: server umount lustre-MDT0000 complete [11557.775632] LustreError: 318907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843145 with bad export cookie 13580620939593230453 [11557.777749] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11557.784340] LustreError: 318907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11558.123272] Lustre: server umount lustre-MDT0001 complete [11568.308346] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11571.726374] Key type lgssc unregistered [11578.151316] LNet: 323049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11578.175558] LNetError: 323049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11659.115436] LNet: Removed LNI 192.168.203.144@tcp [11659.899110] Key type .llcrypt unregistered [11659.902092] Key type ._llcrypt unregistered [11675.845066] Key type ._llcrypt registered [11675.847966] Key type .llcrypt registered [11675.945711] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11690.899454] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11691.788569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11691.812282] alg: No test for adler32 (adler32-zlib) [11692.831233] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11692.975352] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11694.615556] Key type lgssc registered [11695.639081] Lustre: Echo OBD driver; http://www.lustre.org/ [11729.046193] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:08:35 (1787843315) [11736.061734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11736.102628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.325328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11737.354149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11737.454620] Lustre: lustre-MDT0000: new disk, initializing [11737.555955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11737.579323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11742.096975] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11752.208295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11752.335219] Lustre: 326267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11752.396584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11752.407820] Lustre: Skipped 1 previous similar message [11752.493556] Lustre: lustre-MDT0001: new disk, initializing [11752.556695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11752.593961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11752.610317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11757.135796] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11768.345060] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11774.562600] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11782.248449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11782.561713] Lustre: lustre-OST0000: new disk, initializing [11782.566236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11782.584867] Lustre: 328234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11782.684539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11789.531247] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11790.929286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11790.941419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11791.038310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11799.433327] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11816.932846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11816.933430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.940651] LustreError: Skipped 1 previous similar message [11816.952605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11816.958471] Lustre: Skipped 1 previous similar message [11819.123900] Lustre: server umount lustre-OST0000 complete [11824.099399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11824.115646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11824.134386] Lustre: Skipped 1 previous similar message [11824.145693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.173856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.176540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11832.288315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11832.302563] Lustre: Skipped 2 previous similar messages [11836.391964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.397561] Lustre: Skipped 1 previous similar message [11837.407201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11837.702915] Lustre: server umount lustre-MDT0000 complete [11841.887515] LustreError: 326259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843429 with bad export cookie 8628588021723018500 [11841.899167] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11841.901938] LustreError: 326259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11842.267696] Lustre: server umount lustre-MDT0001 complete [11854.166731] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11857.351881] Key type lgssc unregistered [11857.709964] LNet: 330149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11857.728248] LNetError: 330149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11857.756782] LNet: Removed LNI 192.168.203.144@tcp [11858.797489] Key type .llcrypt unregistered [11858.799716] Key type ._llcrypt unregistered [11883.783217] Key type ._llcrypt registered [11883.789518] Key type .llcrypt registered [11883.936123] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11886.244791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11886.394422] alg: No test for adler32 (adler32-zlib) [11887.685846] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11887.913622] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11889.639187] Key type lgssc registered [11890.952549] Lustre: Echo OBD driver; http://www.lustre.org/ [11898.603225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11898.627160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.094663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11904.517165] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11909.603640] LustreError: 331425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.632913] LustreError: 331425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11912.558920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11912.876764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11918.031627] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11925.599426] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11931.188339] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11937.374665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11937.690228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11943.874575] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11951.011965] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11960.891903] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11962.199559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11968.997277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11969.030746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11969.060718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11970.530757] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.533560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11970.566425] Lustre: Skipped 3 previous similar messages [11974.202373] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [11979.333846] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [11979.341439] Lustre: Skipped 3 previous similar messages [11980.255121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11980.529137] Lustre: server umount lustre-MDT0000 complete [11980.773313] LustreError: 331424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.805926] LustreError: 331424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11984.291084] LustreError: 331405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843572 with bad export cookie 9185776467212590598 [11984.299504] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11984.308033] LustreError: 331405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11984.435250] LustreError: 331420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.455417] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [11985.889800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11985.922383] Lustre: Skipped 1 previous similar message [11989.558456] LustreError: 333078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11989.573589] LustreError: 333078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11990.611943] Lustre: server umount lustre-MDT0001 complete [11994.347256] Lustre: server umount lustre-OST0000 complete [12003.831488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12008.422541] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12011.320772] Lustre: 335605:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.144@tcp [12013.464128] Lustre: server umount MGS complete [12022.182226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.551402] LustreError: 336176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.566252] LustreError: 336176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12022.611742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12026.882332] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12032.995026] LustreError: 336176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12033.012634] LustreError: 336176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12035.246183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12035.623537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12039.867356] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12048.326869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12048.640804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12055.340679] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12063.554286] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12065.211110] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12071.768938] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12073.363220] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12081.443937] Lustre: server umount lustre-OST0000 complete [12086.756556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12086.769830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12086.786373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12086.788632] Lustre: Skipped 3 previous similar messages [12095.059723] Lustre: server umount lustre-MDT0000 complete [12097.953836] Lustre: 330737:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843669/real 1787843669] req@000000004c3ed851 x1874689759704832/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787843685 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12098.025171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12098.049121] Lustre: Skipped 1 previous similar message [12098.070615] LustreError: 336502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.105598] LustreError: 336502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12099.766225] LustreError: 336156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843687 with bad export cookie 9185776467212591438 [12099.774452] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12099.787426] LustreError: 336156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12100.153907] Lustre: server umount lustre-MDT0001 complete [12113.682532] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12117.096970] Key type lgssc unregistered [12117.465933] LNet: 339615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12117.476625] LNetError: 339615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12117.506478] LNet: Removed LNI 192.168.203.144@tcp [12118.494162] Key type .llcrypt unregistered [12118.496496] Key type ._llcrypt unregistered [12137.127226] Key type ._llcrypt registered [12137.129575] Key type .llcrypt registered [12137.226041] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12153.105778] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12154.167718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12154.281823] alg: No test for adler32 (adler32-zlib) [12155.267993] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [12155.443381] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12157.111219] Key type lgssc registered [12158.266513] Lustre: Echo OBD driver; http://www.lustre.org/ [12194.091286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12194.115938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12195.387503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12195.431460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12195.550308] Lustre: lustre-MDT0000: new disk, initializing [12195.607178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12195.638159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12200.060717] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12211.736099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12211.862352] Lustre: 342778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12211.916955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12211.920444] Lustre: Skipped 1 previous similar message [12212.004401] Lustre: lustre-MDT0001: new disk, initializing [12212.120535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12212.166144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12212.183721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12216.617318] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12225.647776] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12231.599586] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12238.339499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12238.719344] Lustre: lustre-OST0000: new disk, initializing [12238.723322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12238.729781] Lustre: 344748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12238.839028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12244.242513] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12246.594841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12246.604108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12246.694406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12253.373732] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12258.435705] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12258.572562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12263.162435] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12263.363849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12267.497466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.499905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12267.512479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12267.524694] Lustre: Skipped 1 previous similar message [12267.552338] LustreError: Skipped 1 previous similar message [12271.496415] Lustre: server umount lustre-OST0000 complete [12277.732324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12277.747878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12277.752628] Lustre: Skipped 2 previous similar messages [12282.709034] Lustre: server umount lustre-MDT0000 complete [12286.521031] LustreError: 342772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843874 with bad export cookie 13058694545961887939 [12286.527701] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12286.546690] LustreError: 342772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12286.881977] Lustre: server umount lustre-MDT0001 complete [12296.378477] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:18:02 (1787843882) [12312.976408] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12324.248296] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12360.623346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12360.803552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12360.826877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12360.891959] Lustre: lustre-MDT0000: new disk, initializing [12360.977959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12361.007102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12365.174954] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12375.949179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.068844] Lustre: 349507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12376.146581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12376.152722] Lustre: Skipped 1 previous similar message [12376.268475] Lustre: lustre-MDT0001: new disk, initializing [12376.401488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12376.458671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12376.467490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12380.646028] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12389.698945] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12394.841180] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12401.187986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12401.381800] Lustre: lustre-OST0000: new disk, initializing [12401.385580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12401.390559] Lustre: 351475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12401.469685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12406.919823] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12408.875508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12408.886080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12408.972271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12414.585377] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12445.158371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12445.167451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12445.168791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12445.179896] Lustre: Skipped 2 previous similar messages [12445.188324] Lustre: Skipped 1 previous similar message [12447.005426] Lustre: server umount lustre-OST0000 complete [12453.352752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12453.357598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.374176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12456.678169] Lustre: server umount lustre-MDT0000 complete [12460.132946] LustreError: 349498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844048 with bad export cookie 13058694545961888814 [12460.140555] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12460.144039] LustreError: 349498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12460.513930] Lustre: server umount lustre-MDT0001 complete [12469.995140] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12472.294880] Key type lgssc unregistered [12472.555783] LNet: 353632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12472.564482] LNetError: 353632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12472.591302] LNet: Removed LNI 192.168.203.144@tcp [12473.355924] Key type .llcrypt unregistered [12473.359487] Key type ._llcrypt unregistered [12557.145882] Key type ._llcrypt registered [12557.147383] Key type .llcrypt registered [12557.344224] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12557.833459] Key type .llcrypt unregistered [12557.835889] Key type ._llcrypt unregistered [12640.777361] Key type ._llcrypt registered [12640.782156] Key type .llcrypt registered [12640.880909] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12641.225482] Key type .llcrypt unregistered [12641.228075] Key type ._llcrypt unregistered [12671.799856] Key type ._llcrypt registered [12671.802457] Key type .llcrypt registered [12671.890466] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12672.292326] Key type .llcrypt unregistered [12672.296492] Key type ._llcrypt unregistered [12703.905935] Key type ._llcrypt registered [12703.907955] Key type .llcrypt registered [12703.987780] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12704.260424] Key type .llcrypt unregistered [12704.264983] Key type ._llcrypt unregistered [12776.276194] Key type ._llcrypt registered [12776.281447] Key type .llcrypt registered [12776.368962] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12776.617577] Key type .llcrypt unregistered [12776.621198] Key type ._llcrypt unregistered [12803.049302] Key type ._llcrypt registered [12803.050704] Key type .llcrypt registered [12803.180870] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12803.531384] Key type .llcrypt unregistered [12803.541221] Key type ._llcrypt unregistered [12836.227559] Key type ._llcrypt registered [12836.233614] Key type .llcrypt registered [12836.422169] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12836.886289] Key type .llcrypt unregistered [12836.889529] Key type ._llcrypt unregistered [12868.078350] Key type ._llcrypt registered [12868.086383] Key type .llcrypt registered [12868.224705] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12868.583439] Key type .llcrypt unregistered [12868.590347] Key type ._llcrypt unregistered [12899.069342] Key type ._llcrypt registered [12899.072737] Key type .llcrypt registered [12899.218190] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12899.566478] Key type .llcrypt unregistered [12899.572062] Key type ._llcrypt unregistered [12932.686850] Key type ._llcrypt registered [12932.689568] Key type .llcrypt registered [12932.804238] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12933.280381] Key type .llcrypt unregistered [12933.286296] Key type ._llcrypt unregistered [13005.800521] Key type ._llcrypt registered [13005.803112] Key type .llcrypt registered [13006.001763] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13006.365359] Key type .llcrypt unregistered [13006.369305] Key type ._llcrypt unregistered [13037.577542] Key type ._llcrypt registered [13037.581510] Key type .llcrypt registered [13037.684993] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13038.054441] Key type .llcrypt unregistered [13038.056898] Key type ._llcrypt unregistered [13066.087678] Key type ._llcrypt registered [13066.089458] Key type .llcrypt registered [13066.196660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13066.501352] Key type .llcrypt unregistered [13066.508366] Key type ._llcrypt unregistered [13118.167234] Key type ._llcrypt registered [13118.170688] Key type .llcrypt registered [13118.270643] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13134.103477] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13135.120444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13135.197065] alg: No test for adler32 (adler32-zlib) [13136.342341] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [13136.577430] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [13138.223399] Key type lgssc registered [13139.468667] Lustre: Echo OBD driver; http://www.lustre.org/ [13179.037535] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13179.069634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13180.350748] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13180.372555] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13180.439333] Lustre: tst250fs-MDT0000: new disk, initializing [13180.514056] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13180.536302] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13185.043085] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13195.956181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13196.133669] Lustre: 374018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13196.205687] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13196.210634] Lustre: Skipped 1 previous similar message [13196.288651] Lustre: tst250fs-MDT0001: new disk, initializing [13196.399643] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13196.429080] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13196.439171] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13201.011628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13205.726188] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13212.526209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13212.756696] Lustre: tst250fs-OST0000: new disk, initializing [13212.760948] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13212.767796] Lustre: 375650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13212.915347] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13217.606750] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13221.433065] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13221.454829] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13221.531844] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13227.613292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13227.813899] Lustre: tst250fs-OST0001: new disk, initializing [13227.817789] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13227.825304] Lustre: 376521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13227.994825] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13228.770788] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13228.777202] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13228.906423] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13235.251237] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13241.956327] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13244.236611] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13244.242364] Lustre: Skipped 1 previous similar message [13262.824076] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13262.839105] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.849573] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13264.867813] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13264.868594] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13264.875693] Lustre: Skipped 2 previous similar messages [13264.878617] Lustre: Skipped 2 previous similar messages [13268.805559] Lustre: server umount tst250fs-MDT0000 complete [13272.064625] LustreError: 374012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844859 with bad export cookie 11144120589405581478 [13272.068094] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13272.083972] LustreError: 374012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13272.428213] Lustre: server umount tst250fs-MDT0001 complete [13287.518949] Lustre: server umount tst250fs-OST0000 complete [13301.772984] Lustre: server umount tst250fs-OST0001 complete [13320.194478] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13328.606463] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13361.820809] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:35:48 (1787844948) [13363.624739] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13379.828263] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13390.475566] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13425.246623] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13175 sec ========= 11:36:52 (1787845012) [13426.969874] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:36:53 (1787845013) === [13430.629644] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:36:57 (1787845017) === [13469.110044] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13471.466249] Key type lgssc unregistered [13471.706137] LNet: 385655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13471.713171] LNetError: 385655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13471.748795] LNet: Removed LNI 192.168.203.144@tcp [13472.399141] Key type .llcrypt unregistered [13472.400933] Key type ._llcrypt unregistered