[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509280551 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002300] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.020157] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025082] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029088] *** VALIDATE tmpfs *** [ 0.031121] *** VALIDATE proc *** [ 0.032231] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038013] Speculative Store Bypass: Vulnerable [ 0.041286] debug: unmapping init [mem 0xffffffff99a59000-0xffffffff99a60fff] [ 0.044231] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050020] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052016] ... event mask: 000000070000000f [ 0.053284] rcu: Hierarchical SRCU implementation. [ 0.055460] smp: Bringing up secondary CPUs ... [ 0.056639] x86: Booting SMP configuration: [ 0.057035] .... node #0, CPUs: #1 #2 #3 [ 0.060640] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141022] node 0 deferred pages initialised in 75ms [ 0.144399] devtmpfs: initialized [ 0.146221] x86/mm: Memory block size: 128MB [ 0.148493] gcov: version magic: 0x41383552 [ 0.149585] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154311] pinctrl core: initialized pinctrl subsystem [ 0.157168] [ 0.157827] ************************************************************* [ 0.160014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162011] ** ** [ 0.164014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167015] ** ** [ 0.169010] ** This means that this kernel is built to expose internal ** [ 0.172012] ** IOMMU data structures, which may compromise security on ** [ 0.174011] ** your system. ** [ 0.176012] ** ** [ 0.178011] ** If you see this message and you are not debugging the ** [ 0.180014] ** kernel, report this immediately to your vendor! ** [ 0.183013] ** ** [ 0.185011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187014] ************************************************************* [ 0.191203] NET: Registered protocol family 16 [ 0.192420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202027] cpuidle: using governor menu [ 0.204150] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205399] PCI: Using configuration type 1 for base access [ 0.206095] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214021] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218070] cryptd: max_cpu_qlen set to 1000 [ 0.222258] ACPI: Added _OSI(Module Device) [ 0.224015] ACPI: Added _OSI(Processor Device) [ 0.225027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232110] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238556] ACPI: Interpreter enabled [ 0.240064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242012] ACPI: Using IOAPIC for interrupt routing [ 0.243106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246494] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256463] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272557] acpiphp: Slot [2] registered [ 0.274207] acpiphp: Slot [5] registered [ 0.275128] acpiphp: Slot [6] registered [ 0.277128] acpiphp: Slot [7] registered [ 0.278137] acpiphp: Slot [8] registered [ 0.280127] acpiphp: Slot [9] registered [ 0.282116] acpiphp: Slot [10] registered [ 0.283113] acpiphp: Slot [3] registered [ 0.285106] acpiphp: Slot [4] registered [ 0.286102] acpiphp: Slot [11] registered [ 0.287100] acpiphp: Slot [12] registered [ 0.288096] acpiphp: Slot [13] registered [ 0.289087] acpiphp: Slot [14] registered [ 0.290111] acpiphp: Slot [15] registered [ 0.291087] acpiphp: Slot [16] registered [ 0.292082] acpiphp: Slot [17] registered [ 0.293069] acpiphp: Slot [18] registered [ 0.295084] acpiphp: Slot [19] registered [ 0.296000] acpiphp: Slot [20] registered [ 0.296000] acpiphp: Slot [21] registered [ 0.298128] acpiphp: Slot [22] registered [ 0.300122] acpiphp: Slot [23] registered [ 0.301096] acpiphp: Slot [24] registered [ 0.304115] acpiphp: Slot [25] registered [ 0.305119] acpiphp: Slot [26] registered [ 0.307127] acpiphp: Slot [27] registered [ 0.308100] acpiphp: Slot [28] registered [ 0.310114] acpiphp: Slot [29] registered [ 0.311098] acpiphp: Slot [30] registered [ 0.313099] acpiphp: Slot [31] registered [ 0.314114] PCI host bridge to bus 0000:00 [ 0.315015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.323024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.325022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326247] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.328193] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330392] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346023] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356493] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359908] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366961] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.386026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397061] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.409017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.448015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.468030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.481435] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.489017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.506012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.572462] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.589017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.602019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630970] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.650016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.673016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.685660] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.700017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.707884] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.729020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.743189] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.746443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.748739] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.752418] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.755261] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.760096] iommu: Default domain type: Passthrough [ 0.763646] SCSI subsystem initialized [ 0.765268] ACPI: bus type USB registered [ 0.766125] usbcore: registered new interface driver usbfs [ 0.769166] usbcore: registered new interface driver hub [ 0.771121] usbcore: registered new device driver usb [ 0.773267] pps_core: LinuxPPS API ver. 1 registered [ 0.775014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.777086] PTP clock support registered [ 0.779321] EDAC MC: Ver: 3.0.0 [ 0.782196] PCI: Using ACPI for IRQ routing [ 0.784952] NetLabel: Initializing [ 0.786020] NetLabel: domain hash size = 128 [ 0.788019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.791131] NetLabel: unlabeled traffic allowed by default [ 0.793167] vgaarb: loaded [ 0.795309] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.797018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.804676] clocksource: Switched to clocksource kvm-clock [ 0.914076] VFS: Disk quotas dquot_6.6.0 [ 0.915510] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.918736] *** VALIDATE ramfs *** [ 0.920108] *** VALIDATE hugetlbfs *** [ 0.921857] pnp: PnP ACPI init [ 0.924416] pnp: PnP ACPI: found 6 devices [ 0.980176] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.984196] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.986380] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.989115] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.992035] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.994700] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.997968] NET: Registered protocol family 2 [ 1.000740] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.006416] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.009043] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.014305] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.018178] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023412] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.027066] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.030077] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.032546] NET: Registered protocol family 1 [ 1.035176] RPC: Registered named UNIX socket transport module. [ 1.037773] RPC: Registered udp transport module. [ 1.039855] RPC: Registered tcp transport module. [ 1.041798] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.044254] NET: Registered protocol family 44 [ 1.046048] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.048447] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.050688] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.053254] PCI: CLS 0 bytes, default 64 [ 1.055742] Unpacking initramfs... [ 2.483860] debug: unmapping init [mem 0xffff94437cc54000-0xffff94437ffbffff] [ 2.488481] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.492276] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.495550] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.044334] Initialise system trusted keyrings [ 3.048613] Key type blacklist registered [ 3.051746] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.061294] zbud: loaded [ 3.064886] *** VALIDATE nfs *** [ 3.066294] *** VALIDATE nfs4 *** [ 3.068676] pstore: using deflate compression [ 3.072620] Platform Keyring initialized [ 3.181809] NET: Registered protocol family 38 [ 3.184354] Key type asymmetric registered [ 3.186593] Asymmetric key parser 'x509' registered [ 3.188810] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.191956] io scheduler mq-deadline registered [ 3.194279] io scheduler kyber registered [ 3.196182] io scheduler bfq registered [ 3.198388] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.201817] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.205379] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.208696] ACPI: Power Button [PWRF] [ 3.215636] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.222314] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.236108] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.244046] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.259954] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.287727] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.316467] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.323602] Non-volatile memory driver v1.3 [ 3.325611] Linux agpgart interface v0.103 [ 3.359226] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.364502] vda: detected capacity change from 0 to 74649600 [ 3.383385] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.388032] vdb: detected capacity change from 0 to 1073741824 [ 3.403030] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.407108] vdc: detected capacity change from 0 to 2621440000 [ 3.423690] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.427191] vdd: detected capacity change from 0 to 2621440000 [ 3.444064] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.447474] vde: detected capacity change from 0 to 4294967296 [ 3.462199] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.465667] vdf: detected capacity change from 0 to 4294967296 [ 3.471868] libphy: Fixed MDIO Bus: probed [ 3.478229] usbcore: registered new interface driver usbserial_generic [ 3.481220] usbserial: USB Serial support registered for generic [ 3.483389] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.488396] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.490529] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.493204] mousedev: PS/2 mouse device common for all mice [ 3.496355] rtc_cmos 00:05: RTC can wake from S4 [ 3.499484] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.501047] rtc_cmos 00:05: registered as rtc0 [ 3.505231] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.507723] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.509527] intel_pstate: CPU model not supported [ 3.516226] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.519527] hid: raw HID events driver (C) Jiri Kosina [ 3.522604] usbcore: registered new interface driver usbhid [ 3.525014] usbhid: USB HID core driver [ 3.526954] drop_monitor: Initializing network drop monitor service [ 3.529904] Initializing XFRM netlink socket [ 3.532279] NET: Registered protocol family 10 [ 3.535722] Segment Routing with IPv6 [ 3.537554] NET: Registered protocol family 17 [ 3.540968] mpls_gso: MPLS GSO support [ 3.547164] RAS: Correctable Errors collector initialized. [ 3.549783] AVX version of gcm_enc/dec engaged. [ 3.551473] AES CTR mode by8 optimization enabled [ 3.648632] sched_clock: Marking stable (3648600380, 0)->(4603902442, -955302062) [ 3.654201] registered taskstats version 1 [ 3.656362] Loading compiled-in X.509 certificates [ 3.658268] zswap: loaded using pool lzo/zbud [ 3.686343] Key type big_key registered [ 3.699501] Key type encrypted registered [ 3.701481] ima: No TPM chip found, activating TPM-bypass! [ 3.704369] ima: Allocated hash algorithm: sha1 [ 3.709164] ima: No architecture policies found [ 3.711449] evm: Initialising EVM extended attributes: [ 3.713535] evm: security.selinux [ 3.715161] evm: security.ima [ 3.716390] evm: security.capability [ 3.717850] evm: HMAC attrs: 0x1 [ 3.720474] rtc_cmos 00:05: setting system clock to 2026-07-13 12:49:14 UTC (1783946954) [ 3.726919] debug: unmapping init [mem 0xffffffff9aa03000-0xffffffff9abfffff] [ 3.730095] debug: unmapping init [mem 0xffffffff99782000-0xffffffff99a58fff] [ 3.737111] Write protecting the kernel read-only data: 28672k [ 3.740805] debug: unmapping init [mem 0xffffffff97e03000-0xffffffff97ffffff] [ 3.744204] debug: unmapping init [mem 0xffffffff98714000-0xffffffff987fffff] [ 3.777986] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.786317] systemd[1]: Detected virtualization kvm. [ 3.788035] systemd[1]: Detected architecture x86-64. [ 3.789953] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.816759] systemd[1]: No hostname configured. [ 3.818298] systemd[1]: Set hostname to . [ 3.820089] random: systemd: uninitialized urandom read (16 bytes read) [ 3.822344] systemd[1]: Initializing machine ID from random generator. [ 3.852369] random: ln: uninitialized urandom read (6 bytes read) [ 3.969464] random: systemd: uninitialized urandom read (16 bytes read) [ 3.972333] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.978302] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.983634] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.661786] device-mapper: uevent: version 1.0.3 [ 4.664376] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.391091] random: fast init done [ 5.404143] virtio_net virtio0 ens2: renamed from eth0 [ 5.451987] scsi host0: ata_piix [ 5.475972] scsi host1: ata_piix [ 5.478652] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.481289] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.853090] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.241926] random: crng init done [ 10.246086] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.609676] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.547760] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.416554] SELinux: Disabled at runtime. [ 15.612254] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.633312] systemd[1]: Detected virtualization kvm. [ 15.637564] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.485785] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.491293] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.525786] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.548549] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.564931] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.584926] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.594236] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ 18.100428] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 19.090718] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.313080] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.421292] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.851768] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.947354] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.949401] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 26.599082] NFS: Registering the id_resolver key type [ 26.604618] Key type id_resolver registered [ 26.606475] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-server login: [ 96.757374] libcfs: loading out-of-tree module taints kernel. [ 96.795914] Key type ._llcrypt registered [ 96.798500] Key type .llcrypt registered [ 96.953177] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 119.833530] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 121.622277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.657124] alg: No test for adler32 (adler32-zlib) [ 123.711341] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 125.226462] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 126.959183] Key type lgssc registered [ 129.669687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.776358] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.049895] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 208.169092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.188342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.426529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.465893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.557334] Lustre: lustre-MDT0000: new disk, initializing [ 209.695271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.751622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.889961] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.859893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.986506] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.013773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.018126] Lustre: Skipped 1 previous similar message [ 230.116994] Lustre: lustre-MDT0001: new disk, initializing [ 230.176472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.213948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.237069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.850893] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.913178] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 255.913114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.292781] Lustre: lustre-OST0000: new disk, initializing [ 256.304679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.310990] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.413458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.196938] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.234375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 264.249063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 264.402665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 279.663299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 279.853301] Lustre: lustre-OST0001: new disk, initializing [ 279.857815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 279.865923] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 279.966244] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.273124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.282320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.376732] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 287.739329] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.149614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.242645] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 318.188786] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing check_logdir /tmp/testlogs/ [ 323.897746] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing yml_node [ 328.832987] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 331.947223] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 334.956705] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 336.822108] hrtimer: interrupt took 6617178 ns [ 337.207972] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 13 08:54:44 EDT 2026 [ 357.389586] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 358.857823] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 378.853672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.874063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.908360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.480956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.491632] Lustre: Skipped 7 previous similar messages [ 389.087125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 389.307520] Lustre: server umount lustre-MDT0000 complete [ 393.159820] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947343 with bad export cookie 11648638613026285476 [ 393.161024] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.167196] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 394.722443] LustreError: 12240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.722624] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.726263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 394.747833] LustreError: 12240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 394.768782] Lustre: Skipped 4 previous similar messages [ 398.752286] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.757713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 398.770461] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 398.806359] Lustre: Skipped 2 previous similar messages [ 399.791517] Lustre: server umount lustre-MDT0001 complete [ 404.651064] Lustre: server umount lustre-OST0000 complete [ 409.553721] Lustre: server umount lustre-OST0001 complete [ 417.940227] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 430.063563] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 472.258655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.516455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 472.563474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 472.693631] Lustre: lustre-MDT0000: new disk, initializing [ 472.806174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.834170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 477.908625] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 489.819710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.950589] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 490.001992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 490.006653] Lustre: Skipped 1 previous similar message [ 490.106656] Lustre: lustre-MDT0001: new disk, initializing [ 490.296369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 490.373802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 490.392840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 495.614343] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 506.729953] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 513.290276] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 521.575145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.923052] Lustre: lustre-OST0000: new disk, initializing [ 521.926990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 521.932948] Lustre: 17055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 522.040192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.278527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 523.288167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 523.416071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 529.401576] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 540.648532] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 546.278562] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 546.532771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 552.005731] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 552.290520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 554.464886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 554.473487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.496983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.081839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 559.087836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.544753] Lustre: server umount lustre-OST0000 complete [ 567.266205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 567.298282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.327815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.339616] Lustre: Skipped 2 previous similar messages [ 571.608881] Lustre: server umount lustre-MDT0000 complete [ 574.434418] LustreError: 15097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.473493] LustreError: 15097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 575.954928] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947526 with bad export cookie 11648638613026287009 [ 575.960626] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.973896] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 576.297828] Lustre: server umount lustre-MDT0001 complete [ 583.634448] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:58:51 (1783947531) [ 603.925225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.488506] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.577884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.685292] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 609.779183] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.500284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.421725] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 631.335506] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 637.842114] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 640.360852] Lustre: Failing over lustre-MDT0000 [ 640.482594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.486497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.492613] Lustre: Skipped 2 previous similar messages [ 640.501317] Lustre: Skipped 3 previous similar messages [ 640.596947] Lustre: server umount lustre-MDT0000 complete [ 643.552514] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.580173] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 659.937071] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947594/real 1783947594] req@ffff9442c21c0700 x1870604088285568/t0(0) o400->MGC192.168.203.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783947610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 659.981330] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.018098] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.057526] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 660.205873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.568635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.584756] Lustre: Skipped 1 previous similar message [ 671.642799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 676.848262] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 676.870694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 676.923781] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 677.259781] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 687.874187] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 692.196830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 692.207160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.236575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 698.192046] Lustre: server umount lustre-MDT0000 complete [ 699.382938] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.415928] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 702.560293] LustreError: 19911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947653 with bad export cookie 11648638613026288423 [ 702.575207] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.581076] LustreError: 19911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 703.066391] Lustre: server umount lustre-MDT0001 complete [ 712.575599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.162408] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 728.841105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.968212] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 741.970588] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 747.507974] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 750.148197] Lustre: Failing over lustre-MDT0000 [ 750.717694] Lustre: server umount lustre-MDT0000 complete [ 754.277978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.301211] Lustre: Skipped 2 previous similar messages [ 764.896151] LustreError: 23279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.919300] LustreError: 23279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 770.095312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.234395] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.579941] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 774.637973] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 774.652873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 774.663230] Lustre: Skipped 1 previous similar message [ 774.684232] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 775.389149] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 785.276518] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 789.100288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.112284] Lustre: Skipped 3 previous similar messages [ 795.537744] Lustre: server umount lustre-MDT0000 complete [ 799.697476] LustreError: 23266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947750 with bad export cookie 11648638613026289361 [ 799.708458] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.715972] LustreError: 23266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 800.191291] Lustre: server umount lustre-MDT0001 complete [ 809.469384] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:02:37 (1783947757) [ 827.690919] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 837.635413] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 876.855482] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 888.762891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.235374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 889.307524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 889.452583] Lustre: lustre-MDT0000: new disk, initializing [ 889.576369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.584506] Lustre: Skipped 3 previous similar messages [ 889.602456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 895.273818] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 907.982325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.180903] Lustre: 29659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 908.495976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 913.759738] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 924.250649] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 930.811767] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 938.187093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.664089] Lustre: 31631:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 940.477511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 940.539224] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 940.582579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 940.610232] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 945.967963] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 955.597773] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 961.588736] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 986.597271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.599595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.604019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.611412] Lustre: Skipped 3 previous similar messages [ 986.619524] Lustre: Skipped 2 previous similar messages [ 996.319171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 996.545302] Lustre: server umount lustre-OST0000 complete [ 1014.239186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1014.499430] Lustre: server umount lustre-MDT0000 complete [ 1016.298252] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.313078] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1018.501810] LustreError: 30455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947969 with bad export cookie 11648638613026289767 [ 1018.509285] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.514573] LustreError: 30455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1019.111960] Lustre: server umount lustre-MDT0001 complete [ 1030.523951] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1033.221658] Key type lgssc unregistered [ 1033.530703] LNet: 33614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.535318] LNetError: 33614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.565716] LNet: Removed LNI 192.168.203.128@tcp [ 1034.664184] Key type .llcrypt unregistered [ 1034.668205] Key type ._llcrypt unregistered [ 1062.052254] Key type ._llcrypt registered [ 1062.059522] Key type .llcrypt registered [ 1062.192840] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1063.664966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1063.853596] alg: No test for adler32 (adler32-zlib) [ 1065.001365] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 1065.297234] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1067.127349] Key type lgssc registered [ 1068.520777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1076.809979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1076.854960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.437535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.870843] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1088.992618] LustreError: 35030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.002461] LustreError: 35030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1090.021120] LustreError: 35029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.202266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.763848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.845557] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1106.579927] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1112.546462] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1120.543402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.983570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.184345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1128.357829] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1136.198876] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1138.408559] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1152.393411] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:08:19 (1783948099) [ 1154.641645] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1156.534339] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1164.072824] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:08:31 (1783948111) [ 1173.419845] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:08:41 (1783948121) [ 1174.664419] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1183.162706] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1183.174588] Lustre: Skipped 2 previous similar messages [ 1185.206687] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:08:52 (1783948132) [ 1187.220709] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1187.225436] Lustre: Skipped 2 previous similar messages [ 1195.802266] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1195.808686] Lustre: Skipped 4 previous similar messages [ 1213.011514] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:09:20 (1783948160) [ 1216.396785] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1220.797151] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1229.598685] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:09:37 (1783948177) [ 1231.517960] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1231.531916] Lustre: Skipped 6 previous similar messages [ 1244.768925] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:09:52 (1783948192) [ 1261.537019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1261.541218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.560566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.581812] Lustre: Skipped 1 previous similar message [ 1266.657523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.670441] Lustre: Skipped 2 previous similar messages [ 1271.795575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1271.812408] Lustre: Skipped 1 previous similar message [ 1275.359331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1275.515298] Lustre: server umount lustre-OST0000 complete [ 1282.018204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.031943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1282.035755] Lustre: Skipped 1 previous similar message [ 1282.051659] Lustre: Skipped 1 previous similar message [ 1286.432846] Lustre: server umount lustre-MDT0000 complete [ 1287.142212] LustreError: 35029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.178930] LustreError: 35029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1290.579583] LustreError: 35009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948241 with bad export cookie 16176436757267999713 [ 1290.580910] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1290.596174] LustreError: 35009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1291.311988] Lustre: server umount lustre-MDT0001 complete [ 1306.403207] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1309.921400] Key type lgssc unregistered [ 1310.330148] LNet: 42888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.338706] LNetError: 42888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.357148] LNet: Removed LNI 192.168.203.128@tcp [ 1311.369624] Key type .llcrypt unregistered [ 1311.374345] Key type ._llcrypt unregistered [ 1331.419371] Key type ._llcrypt registered [ 1331.422537] Key type .llcrypt registered [ 1331.504239] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1348.954778] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1350.116792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1350.146199] alg: No test for adler32 (adler32-zlib) [ 1351.308565] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 1351.655763] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1353.439138] Key type lgssc registered [ 1354.954872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1395.981083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1396.013729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.325052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1397.372678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1397.490963] Lustre: lustre-MDT0000: new disk, initializing [ 1397.591960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.623700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1402.217707] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1414.077224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.206070] Lustre: 46044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1414.268947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1414.279234] Lustre: Skipped 1 previous similar message [ 1414.416183] Lustre: lustre-MDT0001: new disk, initializing [ 1414.581557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1414.654502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1414.679632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1419.885663] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1431.289193] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1438.070714] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1445.328927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1445.642657] Lustre: lustre-OST0000: new disk, initializing [ 1445.646538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1445.651433] Lustre: 48013:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1445.755603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.941698] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1455.150510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1455.164749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1455.281047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1462.515456] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1468.214360] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1468.485304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1473.164207] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1473.436976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1476.064251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1476.067640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.074837] LustreError: Skipped 1 previous similar message [ 1476.080928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.095603] Lustre: Skipped 1 previous similar message [ 1476.127898] Lustre: Skipped 1 previous similar message [ 1481.188223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1481.195974] Lustre: Skipped 1 previous similar message [ 1481.931991] Lustre: server umount lustre-OST0000 complete [ 1491.426669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1491.436695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.445349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.461472] Lustre: Skipped 1 previous similar message [ 1491.480892] Lustre: Skipped 1 previous similar message [ 1493.024533] Lustre: server umount lustre-MDT0000 complete [ 1497.547149] LustreError: 46035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948448 with bad export cookie 2671082027492723758 [ 1497.553323] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1497.563276] LustreError: 46035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1498.014349] Lustre: server umount lustre-MDT0001 complete [ 1509.590529] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:14:17 (1783948457) [ 1517.242676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.827089] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.843302] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1517.891754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.175610] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.207294] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1527.285213] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.388657] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.864236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.465293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1538.341335] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1547.244317] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1553.482137] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1560.619095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.936646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.795797] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1576.149392] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1584.204531] LustreError: 53004:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1585.119391] Lustre: Modifying parameter general.timeout=1 in log params [ 1586.899988] Lustre: Modifying parameter general.timeout=3 in log params [ 1586.905609] Lustre: Skipped 1 previous similar message [ 1589.379769] Lustre: Modifying parameter general.timeout=6 in log params [ 1589.383629] Lustre: Skipped 2 previous similar messages [ 1593.645181] Lustre: Modifying parameter general.timeout=10 in log params [ 1593.654986] Lustre: Skipped 3 previous similar messages [ 1601.942422] Lustre: Modifying parameter general.timeout=18 in log params [ 1601.947386] Lustre: Skipped 7 previous similar messages [ 1618.547080] Lustre: Modifying parameter general.timeout=32 in log params [ 1618.553996] Lustre: Skipped 13 previous similar messages [ 1650.927082] Lustre: Modifying parameter general.timeout=58 in log params [ 1650.933254] Lustre: Skipped 25 previous similar messages [ 1716.050673] Lustre: Modifying parameter general.timeout=119 in log params [ 1716.062653] Lustre: Skipped 60 previous similar messages [ 1738.602238] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:18:06 (1783948686) [ 1741.736624] Lustre: 59975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1743.839953] Lustre: 60074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1747.868043] Lustre: 60271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1747.883779] Lustre: 60271:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1752.209136] Lustre: 60487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1752.224908] Lustre: 60487:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1781.430520] Lustre: 61843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1781.466336] LustreError: 61843:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1786.599263] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:18:53 (1783948733) [ 1803.747695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.751560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.766363] Lustre: Skipped 2 previous similar messages [ 1803.774989] Lustre: Skipped 2 previous similar messages [ 1808.871483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.892881] Lustre: Skipped 2 previous similar messages [ 1809.822671] Lustre: server umount lustre-MDT0000 complete [ 1813.990208] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.032292] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1814.917233] LustreError: 59721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948765 with bad export cookie 2671082027492724640 [ 1814.920726] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.941622] LustreError: 59721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1815.531932] Lustre: server umount lustre-MDT0001 complete [ 1831.998097] Lustre: server umount lustre-OST0000 complete [ 1860.611876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.005905] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1861.071278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1861.078978] Lustre: Skipped 33 previous similar messages [ 1861.411542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.394776] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1878.559572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.680854] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1878.700240] Lustre: 64379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1884.657074] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1892.764397] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1899.037669] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1906.331390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1906.648324] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1907.107625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.118384] Lustre: Skipped 1 previous similar message [ 1914.211820] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1922.230392] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1924.846287] Lustre: 66953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1936.864894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1936.876723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.884334] LustreError: Skipped 1 previous similar message [ 1936.888745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1936.899020] Lustre: Skipped 1 previous similar message [ 1939.398670] Lustre: server umount lustre-OST0000 complete [ 1945.570875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.583137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.694893] Lustre: server umount lustre-MDT0000 complete [ 1952.230531] LustreError: 64392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.251026] LustreError: 64392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1953.869838] LustreError: 64371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948904 with bad export cookie 2671082027492727846 [ 1953.879642] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.883125] LustreError: 64371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1954.258877] Lustre: server umount lustre-MDT0001 complete [ 1965.017050] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1967.769639] Key type lgssc unregistered [ 1968.075732] LNet: 68166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.085329] LNetError: 68166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.112138] LNet: Removed LNI 192.168.203.128@tcp [ 1969.030251] Key type .llcrypt unregistered [ 1969.032812] Key type ._llcrypt unregistered [ 1994.357708] Key type ._llcrypt registered [ 1994.368544] Key type .llcrypt registered [ 1994.542557] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1996.156691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.251946] alg: No test for adler32 (adler32-zlib) [ 1997.505558] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 1998.076657] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1999.887471] Key type lgssc registered [ 2001.737535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.685435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.701157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.419160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.180885] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2022.887901] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.900040] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2025.998465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.510702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2031.546854] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2039.099922] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2044.672920] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2051.222926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2051.746240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.710221] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2065.363574] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.302655] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2082.785580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2082.795367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.810478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.810333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.818707] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.278950] Lustre: server umount lustre-MDT0000 complete [ 2088.930438] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.942820] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2089.858076] LustreError: 72264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949040 with bad export cookie 1588933196675990113 [ 2089.865718] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.877319] LustreError: 72264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2090.315452] Lustre: server umount lustre-MDT0001 complete [ 2105.265899] Lustre: server umount lustre-OST0000 complete [ 2131.967143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.302958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2132.360403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2132.602963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.196639] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2146.171967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.272299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2146.315403] Lustre: 74449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2146.898425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2152.459530] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2162.203926] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2168.685850] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2177.934527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2178.240768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2179.021361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.379681] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2194.304889] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2196.073976] Lustre: 77026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2197.924743] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2203.623546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.638694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.653423] Lustre: Skipped 1 previous similar message [ 2203.661727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.667424] Lustre: Skipped 2 previous similar messages [ 2209.257119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2209.264189] Lustre: Skipped 2 previous similar messages [ 2209.522424] Lustre: server umount lustre-OST0000 complete [ 2213.344207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2213.349400] LustreError: Skipped 1 previous similar message [ 2213.354207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.364775] Lustre: Skipped 1 previous similar message [ 2213.370698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.376813] Lustre: Skipped 1 previous similar message [ 2219.498614] LustreError: 75421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.523767] LustreError: 75421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2219.606530] Lustre: server umount lustre-MDT0000 complete [ 2223.557497] LustreError: 76319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949174 with bad export cookie 1588933196675990939 [ 2223.568672] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.568816] LustreError: 76319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2223.985281] Lustre: server umount lustre-MDT0001 complete [ 2234.630215] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2237.455116] Key type lgssc unregistered [ 2237.777776] LNet: 78242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.792973] LNetError: 78242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.811359] LNet: Removed LNI 192.168.203.128@tcp [ 2238.619703] Key type .llcrypt unregistered [ 2238.623296] Key type ._llcrypt unregistered [ 2261.327948] Key type ._llcrypt registered [ 2261.330063] Key type .llcrypt registered [ 2261.419209] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2262.483587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.516648] alg: No test for adler32 (adler32-zlib) [ 2263.597390] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 2263.857669] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2265.583191] Key type lgssc registered [ 2266.815229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2274.445810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2274.474937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.979897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.489793] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2285.549102] LustreError: 79652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.568509] LustreError: 79652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2289.405325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.066781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.528728] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2301.939251] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2307.486914] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2314.017319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2314.743928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.437982] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2328.768448] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2335.742581] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2336.568224] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2338.309181] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2338.314622] Lustre: Skipped 1 previous similar message [ 2340.733373] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2340.737166] Lustre: Skipped 2 previous similar messages [ 2345.183418] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2345.187699] Lustre: Skipped 4 previous similar messages [ 2353.978469] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2353.984898] Lustre: Skipped 8 previous similar messages [ 2370.602101] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2370.609361] Lustre: Skipped 18 previous similar messages [ 2402.899610] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2402.907786] Lustre: Skipped 36 previous similar messages [ 2432.380531] Lustre: 87749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2435.147937] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:29:43 (1783949383) [ 2437.062639] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2439.214681] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:29:46 (1783949386) [ 2443.748156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.759068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2443.779640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.869887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.883609] Lustre: Skipped 2 previous similar messages [ 2449.446499] Lustre: server umount lustre-OST0000 complete [ 2453.987419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.010947] Lustre: Skipped 1 previous similar message [ 2454.024379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.601797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2458.612650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.620574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.225355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.230511] Lustre: Skipped 3 previous similar messages [ 2468.319204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2468.539666] Lustre: server umount lustre-MDT0000 complete [ 2469.344244] LustreError: 79652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.355753] LustreError: 79652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2472.882843] LustreError: 83330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949423 with bad export cookie 2707316695841407180 [ 2472.885029] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.899587] LustreError: 83330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2473.309353] Lustre: server umount lustre-MDT0001 complete [ 2485.512946] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2489.058844] Key type lgssc unregistered [ 2489.321952] LNet: 89000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2489.335655] LNetError: 89000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2489.359629] LNet: Removed LNI 192.168.203.128@tcp [ 2490.352145] Key type .llcrypt unregistered [ 2490.355274] Key type ._llcrypt unregistered [ 2495.028601] Key type ._llcrypt registered [ 2495.031217] Key type .llcrypt registered [ 2495.157689] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2499.703507] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules [ 2500.677091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.814066] alg: No test for adler32 (adler32-zlib) [ 2501.916569] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 2502.238702] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2503.944180] Key type lgssc registered [ 2505.254894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2506.746079] LustreError: 90205:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2525.717262] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2528.761048] LustreError: 90205:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2535.793837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2535.818732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.304939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2542.486384] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2546.663881] LustreError: 91209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.683330] LustreError: 91209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2551.667888] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:31:39 (1783949499) [ 2551.792097] LustreError: 91210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.809637] LustreError: 91241:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2556.821255] LustreError: 91241:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2556.826402] LustreError: 91241:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2557.532485] Lustre: server umount lustre-MDT0000 complete [ 2571.055747] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2574.280558] Key type lgssc unregistered [ 2574.630060] LNet: 92733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2574.643164] LNetError: 92733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2575.719194] LNet: Removed LNI 192.168.203.128@tcp [ 2576.762833] Key type .llcrypt unregistered [ 2576.770269] Key type ._llcrypt unregistered [ 2597.395487] Key type ._llcrypt registered [ 2597.402584] Key type .llcrypt registered [ 2597.496816] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2598.806615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2599.089681] alg: No test for adler32 (adler32-zlib) [ 2600.343653] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 2600.640658] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2602.344275] Key type lgssc registered [ 2603.555399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2611.663589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2611.677780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.164910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.009723] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2622.456696] LustreError: 93968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.491778] LustreError: 93968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2625.340917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.924603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2630.896925] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2639.877709] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.817793] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2652.038266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.612617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.289826] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2667.084930] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2718.244414] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:34:26 (1783949666) [ 2719.918814] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2721.605279] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:34:29 (1783949669) [ 2727.403067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.404330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2727.411181] Lustre: Skipped 2 previous similar messages [ 2727.416798] Lustre: Skipped 2 previous similar messages [ 2731.338543] Lustre: server umount lustre-MDT0000 complete [ 2732.515338] LustreError: 94700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.527114] LustreError: 94700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2734.963093] LustreError: 93947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949685 with bad export cookie 532992026599036493 [ 2734.970673] LustreError: 93947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2734.971464] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.320975] Lustre: server umount lustre-MDT0001 complete [ 2748.640497] Lustre: server umount lustre-OST0000 complete [ 2759.692421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.153469] LustreError: 98242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.378277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.711252] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2765.795260] LustreError: 98243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.895170] LustreError: 98242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.269704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.040176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.248550] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2784.778605] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2789.735607] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2801.453369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.653563] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2801.660961] LustreError: 98233:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2801.680959] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2801.687721] LustreError: 100325:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2801.693697] LustreError: 100325:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2801.698386] LustreError: 100325:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2801.770629] Lustre: server umount lustre-OST0000 complete [ 2801.772404] LustreError: 100325:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2810.749512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.998954] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2811.013517] LustreError: 98232:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2811.044191] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2811.056543] LustreError: 100792:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2811.068649] LustreError: 100792:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2811.075056] LustreError: 100792:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2811.146899] Lustre: server umount lustre-OST0000 complete [ 2811.152266] LustreError: 100792:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2821.453271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.657081] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2821.665446] Lustre: Found index 0 for lustre-OST0000, updating log [ 2821.678307] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2821.772775] Lustre: lustre-OST0000: new disk, initializing [ 2821.776433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2821.783974] Lustre: 101326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2822.044379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.068960] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2831.649193] Lustre: Failing over lustre-OST0000 [ 2831.900345] Lustre: server umount lustre-OST0000 complete [ 2840.033623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2840.044987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.068501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.641789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.645158] Lustre: Skipped 2 previous similar messages [ 2849.760577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.763565] Lustre: Skipped 1 previous similar message [ 2850.273751] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2850.505151] Lustre: server umount lustre-MDT0000 complete [ 2854.323811] LustreError: 98224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949805 with bad export cookie 532992026599039237 [ 2854.327614] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.336335] LustreError: 98224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2854.862291] Lustre: server umount lustre-MDT0001 complete [ 2863.218697] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:36:51 (1783949811) [ 2878.573216] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2888.553381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.961467] LustreError: 103672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.066445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.511778] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2902.292816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.189762] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2910.133730] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2916.379781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2917.042944] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2922.660888] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2927.265937] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2927.270175] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2927.287529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2930.613224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.715416] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2930.827519] Lustre: lustre-OST0001: new disk, initializing [ 2930.843716] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2930.856075] Lustre: 105990:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2931.137515] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2931.155755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2931.156316] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.171328] Lustre: Skipped 2 previous similar messages [ 2934.254827] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2934.282644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2934.310686] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2934.316509] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2934.325460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2936.405133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2936.426472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2936.517525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2937.687782] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2945.180557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2950.895319] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2956.003502] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 2957.289088] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.306317] Lustre: Skipped 3 previous similar messages [ 2960.910372] Lustre: server umount lustre-MDT0001 complete [ 2961.102078] LustreError: 103668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.138677] LustreError: 103668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2972.305934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.371618] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2972.377829] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2972.440936] Lustre: 103660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2972.640991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2972.658542] Lustre: Skipped 5 previous similar messages [ 2974.150920] LustreError: 107534:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2974.168360] LustreError: 107534:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2974.188492] LustreError: 103664:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2974.345512] LustreError: 107544:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -17 [ 2974.366104] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2974.366104] [ 2978.803547] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2979.067722] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2989.229594] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:38:56 (1783949936) [ 3008.072886] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3027.263042] Lustre: 109611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3053.188476] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3056.808223] Lustre: 110748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3151.841353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3151.859784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.890630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.497339] Lustre: server umount lustre-MDT0000 complete [ 3157.987639] LustreError: 105507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3158.009293] LustreError: 105507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3158.044829] LustreError: 106714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950108 with bad export cookie 532992026599040238 [ 3158.051595] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3158.063135] LustreError: 106714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3158.452811] Lustre: server umount lustre-MDT0001 complete [ 3172.441400] Lustre: server umount lustre-OST0000 complete [ 3187.443818] Lustre: server umount lustre-OST0001 complete [ 3193.652844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3204.403578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3213.374194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3224.148557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3245.101282] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3256.202250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.219833] Lustre: lustre-MDT0000: reset Object Index mappings [ 3256.506221] LustreError: 114696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.625411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.634182] Lustre: Skipped 1 previous similar message [ 3261.851548] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3273.058032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.093870] Lustre: lustre-MDT0001: reset Object Index mappings [ 3278.869727] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3281.982678] Lustre: 115851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3288.287693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3295.167785] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3299.216078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3301.238177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3303.474578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.580562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3309.607906] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3309.614948] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3317.439716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3320.777757] Lustre: 117725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3372.549080] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:45:20 (1783950320) [ 3401.699618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3401.706206] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.711757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.741889] Lustre: Skipped 6 previous similar messages [ 3401.757891] Lustre: Skipped 7 previous similar messages [ 3405.654700] Lustre: server umount lustre-MDT0000 complete [ 3406.823739] LustreError: 115437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.855349] LustreError: 115437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3409.614033] LustreError: 114678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950360 with bad export cookie 532992026599087334 [ 3409.618096] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.629192] LustreError: 114678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3411.941169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.220380] Lustre: server umount lustre-MDT0001 complete [ 3426.604375] Lustre: server umount lustre-OST0000 complete [ 3436.581115] Lustre: server umount lustre-OST0001 complete [ 3443.294532] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3453.715089] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3488.967945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3489.171577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3489.268970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3489.414267] Lustre: lustre-MDT0000: new disk, initializing [ 3489.502812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.514047] Lustre: Skipped 3 previous similar messages [ 3489.546391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3494.351367] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3502.686160] Lustre: server umount lustre-MDT0000 complete [ 3513.375354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.807540] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3513.830660] Lustre: 121991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3513.846899] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3518.764399] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3523.606421] Lustre: server umount lustre-MDT0000 complete [ 3531.671444] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:47:59 (1783950479) [ 3533.436914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3535.133831] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:48:03 (1783950483) [ 3552.590781] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3562.606293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.480847] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3577.460104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.572221] Lustre: 124040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3577.626893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3577.630480] Lustre: Skipped 1 previous similar message [ 3577.696791] Lustre: lustre-MDT0001: new disk, initializing [ 3577.879607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3577.893966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3582.361167] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3587.312260] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3593.636797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3593.924993] Lustre: lustre-OST0000: new disk, initializing [ 3593.929295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3593.940081] Lustre: 125618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3595.237929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3595.250572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3595.392263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3599.942737] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3609.892100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3610.102304] Lustre: lustre-OST0001: new disk, initializing [ 3610.105266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3610.111764] Lustre: 126489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3611.656591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3611.660252] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3611.834978] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3616.791113] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3626.225133] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3629.606613] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4022.616399] Lustre: 132910:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4022.623104] Lustre: 132910:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4022.628170] Lustre: 132910:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4022.633640] Lustre: 132910:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4022.643180] Lustre: 132910:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4022.649974] Lustre: 132910:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4332.416512] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:01:19 (1783951279) [ 4349.414358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.422873] Lustre: Skipped 2 previous similar messages [ 4349.423776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.442391] Lustre: Skipped 7 previous similar messages [ 4349.955440] Lustre: server umount lustre-MDT0000 complete [ 4353.717543] LustreError: 124031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783951304 with bad export cookie 532992026599155605 [ 4353.733932] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.736396] LustreError: 124031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4354.528622] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.535916] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.547083] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4354.583508] Lustre: Skipped 3 previous similar messages [ 4358.573654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4358.577597] Lustre: Skipped 2 previous similar messages [ 4368.351472] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4368.962517] Lustre: server umount lustre-MDT0001 complete [ 4387.297838] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4387.525109] Lustre: server umount lustre-OST0000 complete [ 4397.634535] Lustre: server umount lustre-OST0001 complete [ 4404.042712] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4414.163783] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4453.774254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.148984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4454.187662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4454.328429] Lustre: lustre-MDT0000: new disk, initializing [ 4454.548943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.558846] Lustre: Skipped 5 previous similar messages [ 4454.588940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4459.896696] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4471.239023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.409584] Lustre: 135979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4471.455548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4471.459135] Lustre: Skipped 1 previous similar message [ 4471.529983] Lustre: lustre-MDT0001: new disk, initializing [ 4471.674915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4471.692948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4475.853298] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4486.642961] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4492.945355] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4501.117290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.529617] Lustre: lustre-OST0000: new disk, initializing [ 4501.536946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4501.546547] Lustre: 137948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4501.675383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.681103] Lustre: Skipped 1 previous similar message [ 4503.377434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4503.391355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4503.504441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4508.121030] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4517.447924] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4528.968572] Lustre: lustre-MDD0000: changelog on [ 4532.010546] Lustre: lustre-MDD0001: changelog on [ 6622.616958] Lustre: lustre-MDD0001: changelog off [ 6624.803310] Lustre: 135986:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6625.651293] Lustre: lustre-MDD0000: changelog off [ 6634.472731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.491202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6634.508269] Lustre: Skipped 4 previous similar messages [ 6639.594402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6639.621298] Lustre: Skipped 1 previous similar message [ 6639.923971] Lustre: server umount lustre-OST0000 complete [ 6644.721887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.726200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6644.753200] Lustre: Skipped 2 previous similar messages [ 6644.766577] Lustre: Skipped 1 previous similar message [ 6650.791949] Lustre: server umount lustre-MDT0000 complete [ 6654.294768] LustreError: 136784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953605 with bad export cookie 532992026599977531 [ 6654.295203] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6654.317099] LustreError: 136784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6654.686169] Lustre: server umount lustre-MDT0001 complete [ 6666.791446] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6671.241693] Key type lgssc unregistered [ 6671.568578] LNet: 147126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6671.582336] LNetError: 147126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6671.605512] LNet: Removed LNI 192.168.203.128@tcp [ 6672.381190] Key type .llcrypt unregistered [ 6672.384941] Key type ._llcrypt unregistered [ 6696.916371] Key type ._llcrypt registered [ 6696.922511] Key type .llcrypt registered [ 6697.030544] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6713.364482] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6715.161760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6715.195919] alg: No test for adler32 (adler32-zlib) [ 6716.366378] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 6716.718236] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6718.433110] Key type lgssc registered [ 6719.822457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6760.275304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6760.309342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.775503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6761.847226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6761.989669] Lustre: lustre-MDT0000: new disk, initializing [ 6762.177109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.226203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6768.353959] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6782.833241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.072947] Lustre: 150328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6783.163749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6783.169222] Lustre: Skipped 1 previous similar message [ 6783.294343] Lustre: lustre-MDT0001: new disk, initializing [ 6783.447105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6783.513782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6783.535837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6789.037626] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6799.468823] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6805.468953] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6813.153442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6813.553227] Lustre: lustre-OST0000: new disk, initializing [ 6813.557669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6813.571870] Lustre: 152298:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6813.691106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.154381] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6824.039833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6824.061589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6824.253966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6831.289983] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6836.884907] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_module obdecho/obdecho [ 6838.821862] LustreError: 153300:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6838.838888] LustreError: 153300:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6840.559586] LustreError: 153396:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6855.145030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.149068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6855.154125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6855.172932] Lustre: Skipped 1 previous similar message [ 6860.171997] Lustre: server umount lustre-MDT0000 complete [ 6864.814231] LustreError: 152308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953815 with bad export cookie 576708304485420303 [ 6864.818838] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.832735] LustreError: 152308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6865.352285] Lustre: server umount lustre-MDT0001 complete [ 6880.594692] Lustre: server umount lustre-OST0000 complete [ 6906.783077] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6918.319170] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6954.896252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.147727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6955.207955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6955.302832] Lustre: lustre-MDT0000: new disk, initializing [ 6955.390785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6955.417720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6959.958834] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6971.358410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.451364] Lustre: 157113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6971.506378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6971.510365] Lustre: Skipped 1 previous similar message [ 6971.578802] Lustre: lustre-MDT0001: new disk, initializing [ 6971.678048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6971.731607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6971.753444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6977.048669] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6987.697651] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6994.296595] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7001.597126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.994541] Lustre: lustre-OST0000: new disk, initializing [ 7001.997991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7002.011495] Lustre: 159082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7002.102782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.820239] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7010.907882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7010.917031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7011.001633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7018.443764] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.405965] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7024.705269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7029.862771] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7030.129347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7036.906639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7036.916272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.920231] LustreError: Skipped 1 previous similar message [ 7036.954238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7036.957236] Lustre: Skipped 2 previous similar messages [ 7036.997167] Lustre: Skipped 3 previous similar messages [ 7038.238652] Lustre: server umount lustre-OST0000 complete [ 7043.566977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7043.583874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.602098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7047.136537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7047.146435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.052496] Lustre: server umount lustre-MDT0000 complete [ 7052.256902] LustreError: 157122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7052.285470] LustreError: 157122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7053.661502] LustreError: 157909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954004 with bad export cookie 576708304485421262 [ 7053.663906] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.676863] LustreError: 157909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7054.345895] Lustre: server umount lustre-MDT0001 complete [ 7058.902491] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:46:46 (1783954006) [ 7061.090593] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7063.416524] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:46:50 (1783954010) [ 7073.287630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.739266] LustreError: 161494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.811733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.892569] LustreError: 161495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.120640] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7084.009969] LustreError: 161494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.739560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.123482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7089.133651] Lustre: Skipped 1 previous similar message [ 7094.980936] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7103.362768] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7109.312725] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7117.151399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7117.831762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.846794] Lustre: Skipped 1 previous similar message [ 7126.071684] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7134.731570] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7145.443681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.447304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7145.456839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7145.462800] Lustre: Skipped 1 previous similar message [ 7147.280069] Lustre: server umount lustre-MDT0000 complete [ 7150.562443] LustreError: 163990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7150.591124] LustreError: 163990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7152.086748] LustreError: 161476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954102 with bad export cookie 576708304485422144 [ 7152.102902] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7152.123854] LustreError: 161476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7155.681649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.700990] Lustre: Skipped 1 previous similar message [ 7158.722609] Lustre: server umount lustre-MDT0001 complete [ 7165.276408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7179.049325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.361754] LustreError: 165338:0:(llog.c:1655:llog_backup()) MGC192.168.203.128@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7179.376278] Lustre: 165338:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7181.283874] LustreError: 165343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.309574] LustreError: 165343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7197.900499] LustreError: 165344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7197.917418] LustreError: 165344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7232.481548] LustreError: 165345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.506589] LustreError: 165345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7240.159437] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7240.172162] Lustre: 165338:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7281.161149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7285.984738] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7286.250066] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7294.468259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.486990] LustreError: 165362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7296.497536] LustreError: 165362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 7301.789949] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7302.140965] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7310.423262] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7312.216439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7318.394551] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7320.360371] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7342.372796] Lustre: server umount lustre-MDT0000 complete [ 7343.090656] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.101332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7343.113164] Lustre: Skipped 1 previous similar message [ 7346.405747] LustreError: 164279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954297 with bad export cookie 576708304485428598 [ 7346.408388] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.428465] LustreError: 164279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7347.040947] Lustre: server umount lustre-MDT0001 complete [ 7360.509506] Lustre: server umount lustre-OST0000 complete [ 7385.442315] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 7395.427420] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7440.295190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.514521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7440.552174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7440.630790] Lustre: lustre-MDT0000: new disk, initializing [ 7440.739684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.744801] Lustre: Skipped 1 previous similar message [ 7440.779419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7445.462823] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7458.667839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7458.875717] Lustre: 170444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7459.253839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7465.654348] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7475.898609] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7481.913827] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7489.689144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.130746] Lustre: lustre-OST0000: new disk, initializing [ 7490.141716] Lustre: Skipped 1 previous similar message [ 7490.145660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7490.152664] Lustre: Skipped 2 previous similar messages [ 7490.166393] Lustre: 172414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7496.840138] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7499.369584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7499.377364] Lustre: Skipped 1 previous similar message [ 7499.383498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7499.518170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7507.572779] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7512.834416] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7513.092970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7518.095972] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7518.328203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7525.354182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.365997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7525.380577] Lustre: Skipped 1 previous similar message [ 7525.391956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.421621] Lustre: Skipped 7 previous similar messages [ 7526.499726] Lustre: server umount lustre-OST0000 complete [ 7536.927059] Lustre: server umount lustre-MDT0000 complete [ 7540.712056] LustreError: 171621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.724733] LustreError: 171621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7541.427683] LustreError: 170435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954492 with bad export cookie 576708304485445426 [ 7541.432054] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7541.448032] LustreError: 170435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7541.799324] Lustre: server umount lustre-MDT0001 complete [ 7551.225410] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:54:58 (1783954498) [ 7558.815049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7564.167829] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7573.752800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.971302] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7586.724236] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7593.182485] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7601.151369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.575846] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7616.117197] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7629.551774] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:56:17 (1783954577) [ 7634.402159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.405501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7634.428729] Lustre: Skipped 4 previous similar messages [ 7634.436715] Lustre: Skipped 3 previous similar messages [ 7640.134822] Lustre: server umount lustre-OST0000 complete [ 7650.542906] Lustre: server umount lustre-MDT0000 complete [ 7654.705955] LustreError: 174808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954605 with bad export cookie 576708304485446315 [ 7654.709078] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7654.718913] LustreError: 174808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7655.216532] Lustre: server umount lustre-MDT0001 complete [ 7666.025978] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 7669.611660] Key type lgssc unregistered [ 7670.016945] LNet: 178554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7670.034188] LNetError: 178554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7670.072182] LNet: Removed LNI 192.168.203.128@tcp [ 7671.671209] Key type .llcrypt unregistered [ 7671.673570] Key type ._llcrypt unregistered [ 7693.768391] Key type ._llcrypt registered [ 7693.781139] Key type .llcrypt registered [ 7693.941556] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7695.138599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7695.181222] alg: No test for adler32 (adler32-zlib) [ 7696.187888] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 7696.472119] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7698.183198] Key type lgssc registered [ 7699.429060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7707.058952] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7707.078820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.655496] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7727.775386] LustreError: 179798:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 7732.895587] LustreError: 179798:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7732.918227] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7732.924802] LustreError: MGC192.168.203.128@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7732.940445] LustreError: 179798:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7732.958729] LustreError: 179798:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7732.971831] LustreError: 179798:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7733.051761] Lustre: server umount lustre-OST0000 complete [ 7733.056494] LustreError: 179798:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7743.202899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.860961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.582603] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7755.254786] LustreError: 180298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7755.287444] LustreError: 180298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7759.311742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7759.811339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7764.837323] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7772.209625] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7778.454338] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7785.624563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7785.859184] LustreError: 182117:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7785.876532] Lustre: 182117:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.128@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7786.048946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7792.431822] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7801.230591] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7804.518776] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.540470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7804.560350] Lustre: Skipped 2 previous similar messages [ 7804.586613] Lustre: Skipped 2 previous similar messages [ 7809.873316] Lustre: server umount lustre-MDT0000 complete [ 7810.022841] LustreError: 181287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.056358] LustreError: 181287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7814.530573] LustreError: 180280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954765 with bad export cookie 11225572128033180586 [ 7814.546606] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7814.840295] LustreError: 181029:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000818ff234 x1870612027962240/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7815.136780] Lustre: server umount lustre-MDT0001 complete [ 7830.077269] Lustre: server umount lustre-OST0000 complete [ 7841.899284] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 7852.184535] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7894.197936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.445102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7894.480398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7894.611734] Lustre: lustre-MDT0000: new disk, initializing [ 7894.686697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7894.708773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7899.533346] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7911.392190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.557579] Lustre: 185641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7911.635344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7911.642126] Lustre: Skipped 1 previous similar message [ 7911.769344] Lustre: lustre-MDT0001: new disk, initializing [ 7911.886364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7911.937311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7911.950345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7917.744672] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7927.700720] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7933.916938] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7940.699279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7941.021792] Lustre: lustre-OST0000: new disk, initializing [ 7941.030598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7941.047678] Lustre: 187620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7941.172365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7947.832725] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7947.881050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7947.885933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7948.036429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7957.502459] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7962.875435] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7963.123176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7967.731938] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7967.953542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7973.859991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7973.865679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7973.869049] LustreError: Skipped 1 previous similar message [ 7973.882544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7973.897889] Lustre: Skipped 1 previous similar message [ 7976.125407] Lustre: server umount lustre-OST0000 complete [ 7983.590735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7983.603344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.624591] Lustre: Skipped 1 previous similar message [ 7983.630182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7989.216745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7989.224439] Lustre: Skipped 3 previous similar messages [ 7993.315566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7993.329063] Lustre: Skipped 1 previous similar message [ 7994.336374] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7994.465746] Lustre: server umount lustre-MDT0000 complete [ 7998.432781] LustreError: 185653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.448277] LustreError: 185653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7998.639567] LustreError: 185633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954949 with bad export cookie 11225572128033181307 [ 7998.646404] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.654578] LustreError: 185633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7999.016271] Lustre: server umount lustre-MDT0001 complete [ 8007.837489] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:02:35 (1783954955) [ 8026.702363] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 8029.419499] Key type lgssc unregistered [ 8029.680626] LNet: 190567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8029.689585] LNetError: 190567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8029.700818] LNet: Removed LNI 192.168.203.128@tcp [ 8030.468709] Key type .llcrypt unregistered [ 8030.471333] Key type ._llcrypt unregistered [ 8050.898710] Key type ._llcrypt registered [ 8050.901224] Key type .llcrypt registered [ 8050.996125] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8052.008256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8052.084077] alg: No test for adler32 (adler32-zlib) [ 8053.137363] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 8053.457555] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 8055.175222] Key type lgssc registered [ 8056.365323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8063.722869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8063.748214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.171540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8069.740797] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8074.733182] LustreError: 191785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8074.769740] LustreError: 191785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8078.438511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8078.997547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8084.142638] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8092.135410] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8099.011835] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8107.531415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.866405] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8107.870864] LustreError: 193653:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8107.883076] Lustre: 193653:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.128@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8107.983909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.967518] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8121.868687] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8124.898386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8124.906975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.916277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8128.481696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8128.495462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8128.515863] Lustre: Skipped 1 previous similar message [ 8130.560243] Lustre: server umount lustre-MDT0000 complete [ 8133.607384] LustreError: 191781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8133.626274] LustreError: 191781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8134.160575] LustreError: 192521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955084 with bad export cookie 6338901256957708633 [ 8134.161461] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8134.184510] LustreError: 192521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8134.540884] Lustre: server umount lustre-MDT0001 complete [ 8149.088358] Lustre: server umount lustre-OST0000 complete [ 8157.790324] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8167.573204] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8206.307348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8206.610709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8206.650860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8206.807133] Lustre: lustre-MDT0000: new disk, initializing [ 8206.958401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8206.975902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8212.329711] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8227.771990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.987253] Lustre: 197176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8228.068196] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8228.072199] Lustre: Skipped 1 previous similar message [ 8228.082576] LustreError: 197959:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8228.091595] LustreError: 197959:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8228.107747] Lustre: 197959:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8228.161122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8228.169938] Lustre: Skipped 1 previous similar message [ 8228.234675] Lustre: lustre-MDT0001: new disk, initializing [ 8228.336589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8228.401062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8228.413818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8234.609847] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8245.913797] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8252.800491] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8260.207632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8260.604466] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8260.609114] Lustre: Skipped 1 previous similar message [ 8260.621024] LustreError: 199119:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8260.638516] LustreError: 199119:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8260.649219] Lustre: 199119:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8260.686291] Lustre: 199119:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8260.733543] Lustre: lustre-OST0000: new disk, initializing [ 8260.747993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8260.760904] Lustre: 199146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8260.883122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8269.480053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8269.498709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8269.664284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8269.847722] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8281.104431] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8286.714575] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8287.004645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8291.647284] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8291.880312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8295.402893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8295.405100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.411211] LustreError: Skipped 1 previous similar message [ 8295.412375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8295.412381] Lustre: Skipped 2 previous similar messages [ 8295.452336] Lustre: Skipped 1 previous similar message [ 8300.523592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8300.531978] Lustre: Skipped 2 previous similar messages [ 8305.642043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8305.653324] Lustre: Skipped 1 previous similar message [ 8308.191481] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8308.390139] Lustre: server umount lustre-OST0000 complete [ 8315.361804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8315.372834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.390827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.401243] Lustre: server umount lustre-MDT0000 complete [ 8320.992813] LustreError: 197186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8321.013552] LustreError: 197186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8322.247759] LustreError: 199143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955273 with bad export cookie 6338901256957709347 [ 8322.251238] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8322.259123] LustreError: 199143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8322.633592] Lustre: server umount lustre-MDT0001 complete [ 8331.241963] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:07:59 (1783955279) [ 8347.561793] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8357.271043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8357.664911] LustreError: 202277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.718646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.848436] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8362.977930] LustreError: 202278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8369.917297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.771952] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8379.207784] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8385.878196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.113733] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8403.572184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.841584] Lustre: lustre-OST0001: new disk, initializing [ 8403.851098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8403.857724] Lustre: 204586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8403.972525] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8403.976148] Lustre: Skipped 2 previous similar messages [ 8410.672817] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8413.251353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8413.269437] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8413.351913] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8421.822321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8430.154411] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8433.050160] Lustre: DEBUG MARKER: ADD OST3 [ 8438.443643] loop: module loaded [ 8438.539356] loop0: detected capacity change from 0 to 204800000 [ 8438.775749] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8438.787243] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8438.807518] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8445.975234] loop0: detected capacity change from 0 to 204800000 [ 8446.588562] Lustre: DEBUG MARKER: STOP OST3 [ 8451.927293] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8452.160695] Lustre: lustre-OST0003: new disk, initializing [ 8452.167685] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8452.174396] Lustre: 206579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8453.853643] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8454.008179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8454.021315] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8454.076247] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8455.910068] LustreError: 206759:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8456.074291] Lustre: Failing over lustre-OST0003 [ 8456.305881] Lustre: server umount lustre-OST0003 complete [ 8458.482384] LustreError: 205116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.499478] LustreError: 205116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8459.247858] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8459.250424] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8459.276202] Lustre: Skipped 1 previous similar message [ 8460.022038] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8469.507683] Lustre: DEBUG MARKER: START OST3 again [ 8474.602090] LustreError: 203765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8474.627915] LustreError: 203765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8475.968840] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8476.279149] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8476.290527] Lustre: Skipped 1 previous similar message [ 8476.305753] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8477.379855] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8477.865955] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8477.866109] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8484.321430] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8506.163783] Lustre: Failing over lustre-OST0003 [ 8506.263424] Lustre: server umount lustre-OST0003 complete [ 8508.896299] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8508.906355] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.909595] LustreError: Skipped 1 previous similar message [ 8508.911587] LustreError: 205116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8508.946147] Lustre: Skipped 2 previous similar messages [ 8509.006996] LustreError: 205116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8525.798107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8525.801134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8525.824747] Lustre: Skipped 2 previous similar messages [ 8525.828472] Lustre: Skipped 2 previous similar messages [ 8530.746178] Lustre: server umount lustre-MDT0000 complete [ 8535.050864] LustreError: 202259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955485 with bad export cookie 6338901256957710236 [ 8535.055794] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8535.074828] LustreError: 202259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8541.437465] Lustre: server umount lustre-MDT0001 complete [ 8545.727775] Lustre: server umount lustre-OST0000 complete [ 8549.725404] Lustre: server umount lustre-OST0001 complete [ 8557.116041] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8567.307517] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8608.470747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.689783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8608.732047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8608.845743] Lustre: lustre-MDT0000: new disk, initializing [ 8608.967686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.997494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8613.968859] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8624.688507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8624.839188] Lustre: 212045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8624.933340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8624.945235] Lustre: Skipped 1 previous similar message [ 8625.033986] Lustre: lustre-MDT0001: new disk, initializing [ 8625.172343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8625.187638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8629.981553] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8639.540867] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8646.365368] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8654.425845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8654.745694] Lustre: 214016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8656.281134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8656.435430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8660.423475] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8671.151484] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8676.283340] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8676.571533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8681.893505] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8682.157611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8687.073578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8687.083562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.095017] Lustre: Skipped 3 previous similar messages [ 8687.100264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8687.108524] Lustre: Skipped 7 previous similar messages [ 8690.358571] Lustre: server umount lustre-OST0000 complete [ 8708.063137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8708.374443] Lustre: server umount lustre-MDT0000 complete [ 8711.648920] LustreError: 212926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8711.662530] LustreError: 212926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8711.890419] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955662 with bad export cookie 6338901256957712532 [ 8711.895157] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8711.908128] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8712.250730] Lustre: server umount lustre-MDT0001 complete [ 8715.938516] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:14:23 (1783955663) [ 8732.711484] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8744.070810] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8779.949112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.218153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8780.250074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8780.255434] Lustre: Skipped 1 previous similar message [ 8780.344456] Lustre: lustre-MDT0000: new disk, initializing [ 8780.347171] Lustre: Skipped 1 previous similar message [ 8780.444987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8780.454647] Lustre: Skipped 1 previous similar message [ 8784.369551] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8794.536153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8794.679623] Lustre: 218629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8794.910223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8798.456311] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8806.952233] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8811.942795] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8817.752973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8817.997702] Lustre: 220601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8819.333303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8824.262581] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8833.547771] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8837.898430] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8838.098518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8842.202829] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8842.394872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8844.793879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8844.794620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8844.796417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8844.796424] Lustre: Skipped 7 previous similar messages [ 8844.805472] Lustre: Skipped 4 previous similar messages [ 8844.843872] LustreError: Skipped 2 previous similar messages [ 8850.417612] Lustre: server umount lustre-OST0000 complete [ 8868.319181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8868.677295] Lustre: server umount lustre-MDT0000 complete [ 8870.383620] LustreError: 218637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.396337] LustreError: 218637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8872.707674] LustreError: 218620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955823 with bad export cookie 6338901256957713414 [ 8872.716819] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8872.725210] LustreError: 218620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8873.030464] Lustre: server umount lustre-MDT0001 complete [ 8880.827835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.356533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8881.365419] Lustre: Skipped 5 previous similar messages [ 8885.379733] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8893.541329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.893032] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8904.985508] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8910.145498] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8917.045505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.468614] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8930.857204] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8942.069223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.625471] Lustre: server umount lustre-OST0000 complete [ 8957.030093] Lustre: server umount lustre-MDT0000 complete [ 8961.063944] LustreError: 222798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955911 with bad export cookie 6338901256957714296 [ 8961.076963] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.569334] Lustre: server umount lustre-MDT0001 complete [ 9069.029417] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 9071.801438] Key type lgssc unregistered [ 9072.113890] LNet: 226476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9072.119185] LNetError: 226476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9073.204798] LNet: Removed LNI 192.168.203.128@tcp [ 9074.115144] Key type .llcrypt unregistered [ 9074.116947] Key type ._llcrypt unregistered [ 9099.692791] Key type ._llcrypt registered [ 9099.694978] Key type .llcrypt registered [ 9099.794575] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 9114.118938] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9115.242574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9115.483893] alg: No test for adler32 (adler32-zlib) [ 9116.519984] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 9116.815901] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 9118.503167] Key type lgssc registered [ 9119.709725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9153.885939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9153.910613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9155.097861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9155.129204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9155.206276] Lustre: lustre-MDT0000: new disk, initializing [ 9155.270519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9155.285634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9159.335389] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9170.768757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.901661] Lustre: 229827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9170.957472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9170.961342] Lustre: Skipped 1 previous similar message [ 9171.040493] Lustre: lustre-MDT0001: new disk, initializing [ 9171.136397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.171127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9171.189294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9176.485154] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9185.690887] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9191.259624] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9198.774565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9199.195910] Lustre: lustre-OST0000: new disk, initializing [ 9199.206215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9199.219931] Lustre: 231797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9199.337068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9206.022349] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9207.356076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9207.372570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9207.523252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9215.334691] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9220.534205] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9220.793689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9225.795088] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9226.115775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9228.261974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9228.267692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.275375] LustreError: Skipped 1 previous similar message [ 9228.281768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9228.292058] Lustre: Skipped 1 previous similar message [ 9228.312641] Lustre: Skipped 1 previous similar message [ 9233.386383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9233.391102] Lustre: Skipped 1 previous similar message [ 9234.512522] Lustre: server umount lustre-OST0000 complete [ 9237.987770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9237.992534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.001599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9243.620049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9243.630512] Lustre: Skipped 3 previous similar messages [ 9248.738985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9248.742796] Lustre: Skipped 1 previous similar message [ 9252.319206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9252.558040] Lustre: server umount lustre-MDT0000 complete [ 9256.759933] LustreError: 229822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956207 with bad export cookie 10502610466804300249 [ 9256.764769] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9256.775495] LustreError: 229822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9257.233082] Lustre: server umount lustre-MDT0001 complete [ 9273.854106] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9282.465490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.718837] LustreError: 234732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9282.732424] LustreError: 234732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9282.753141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9286.912894] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9288.163153] LustreError: 234733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9292.263966] LustreError: 234732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9294.611678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9294.940992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9298.272709] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9300.591274] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9306.057289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9306.327853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.242407] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9320.994379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9321.226986] Lustre: lustre-OST0001: new disk, initializing [ 9321.234433] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9321.242198] Lustre: 237040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9327.249391] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9330.288252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9330.309434] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9330.448317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9337.208500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9347.414976] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9360.195529] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:25:08 (1783956308) [ 9366.496704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9366.501607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9366.503138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9366.524256] Lustre: Skipped 3 previous similar messages [ 9368.298222] Lustre: server umount lustre-MDT0000 complete [ 9371.616396] LustreError: 235828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.635306] LustreError: 235828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9371.747616] LustreError: 234712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956322 with bad export cookie 10502610466804301131 [ 9371.760924] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9371.761712] LustreError: 234712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9372.202932] Lustre: server umount lustre-MDT0001 complete [ 9386.180651] Lustre: server umount lustre-OST0000 complete [ 9400.394806] Lustre: server umount lustre-OST0001 complete [ 9406.243717] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 9416.201623] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9447.671844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.841970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9447.862934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9447.917524] Lustre: lustre-MDT0000: new disk, initializing [ 9447.998297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9448.001849] Lustre: Skipped 1 previous similar message [ 9448.011350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9451.456222] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9460.573280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9460.680612] Lustre: 240993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9460.722044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9460.732874] Lustre: Skipped 1 previous similar message [ 9460.812679] Lustre: lustre-MDT0001: new disk, initializing [ 9460.939192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9460.950789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9464.729698] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9472.289887] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9476.060502] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9481.517814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9481.763584] Lustre: lustre-OST0000: new disk, initializing [ 9481.767463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9481.773875] Lustre: 242960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9481.863742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9481.872574] Lustre: Skipped 1 previous similar message [ 9483.601786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9483.608106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9483.691106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9486.273889] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9493.563296] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9497.284943] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9497.429394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9500.492754] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9500.632692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9504.227269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9504.227688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.228558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9504.228564] Lustre: Skipped 3 previous similar messages [ 9504.233300] LustreError: Skipped 1 previous similar message [ 9504.255110] Lustre: Skipped 2 previous similar messages [ 9508.097178] Lustre: server umount lustre-OST0000 complete [ 9512.416720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9512.423733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9525.215634] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9525.375529] Lustre: server umount lustre-MDT0000 complete [ 9528.746034] LustreError: 241786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956479 with bad export cookie 10502610466804303266 [ 9528.747796] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9528.750569] LustreError: 241786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9529.050513] Lustre: server umount lustre-MDT0001 complete [ 9537.079364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9537.484404] LustreError: 245177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9540.931991] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9547.748949] LustreError: 245177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9547.785097] LustreError: 245177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9549.277701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.514663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9549.525988] Lustre: Skipped 1 previous similar message [ 9553.426935] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9560.501687] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9566.053273] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9572.104078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9578.871993] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9586.381977] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9736.681953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9736.692161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9736.701954] Lustre: Skipped 2 previous similar messages [ 9736.706442] Lustre: Skipped 8 previous similar messages [ 9736.716720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9740.270470] Lustre: server umount lustre-OST0000 complete [ 9751.112780] Lustre: server umount lustre-MDT0000 complete [ 9752.043457] LustreError: 245904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.059092] LustreError: 245904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9754.755569] LustreError: 245157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956705 with bad export cookie 10502610466804304148 [ 9754.763598] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9754.770095] LustreError: 245157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9755.094550] Lustre: server umount lustre-MDT0001 complete [ 9857.374365] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 9860.467441] Key type lgssc unregistered [ 9860.764933] LNet: 248782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9860.777709] LNetError: 248782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9860.794986] LNet: Removed LNI 192.168.203.128@tcp [ 9861.700157] Key type .llcrypt unregistered [ 9861.707855] Key type ._llcrypt unregistered [ 9890.047212] Key type ._llcrypt registered [ 9890.050816] Key type .llcrypt registered [ 9890.230813] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 9905.284606] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9906.083651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9906.251549] alg: No test for adler32 (adler32-zlib) [ 9907.285476] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 9907.520136] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 9909.175201] Key type lgssc registered [ 9910.073864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9955.501480] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9969.004437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9969.036949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.421113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9970.466501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9970.620769] Lustre: lustre-MDT0000: new disk, initializing [ 9970.765492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9970.787107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9976.328877] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9988.127274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9988.385802] Lustre: 252884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9988.469267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9988.475769] Lustre: Skipped 1 previous similar message [ 9988.606864] Lustre: lustre-MDT0001: new disk, initializing [ 9988.782628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9988.854240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9988.876259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9994.143136] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9999.679410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10007.802683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10008.165461] Lustre: lustre-OST0000: new disk, initializing [10008.171965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10008.181502] Lustre: 254516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10008.305067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10015.234728] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10015.329025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10015.344213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10015.477553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10027.166668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10027.367742] Lustre: lustre-OST0001: new disk, initializing [10027.378649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10027.387988] Lustre: 255388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10027.522927] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10033.947423] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10035.900940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10035.931652] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10036.102247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10045.780527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10057.551231] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10070.496750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10070.504882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.520583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.035539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.039138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.058663] Lustre: Skipped 2 previous similar messages [10076.807711] Lustre: server umount lustre-MDT0000 complete [10080.401321] LustreError: 252876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957031 with bad export cookie 18331783006981815507 [10080.411807] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10080.423617] LustreError: 252876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10080.785638] Lustre: server umount lustre-MDT0001 complete [10094.987102] Lustre: server umount lustre-OST0000 complete [10098.465944] Lustre: 250365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783957033/real 1783957033] req@ffff9442cdfe6300 x1870614346413312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783957049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10098.505916] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10098.780571] Lustre: server umount lustre-OST0001 complete [10113.022629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10123.127263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10131.394946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10141.235588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10163.495956] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10174.950349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10174.994365] Lustre: lustre-MDT0000: reset Object Index mappings [10175.594062] LustreError: 260397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.612062] LustreError: 260397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10175.698248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10180.605736] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10181.090562] LustreError: 260398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10186.227176] LustreError: 260397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.471831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.484787] Lustre: lustre-MDT0001: reset Object Index mappings [10190.980655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10196.116948] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10199.998593] Lustre: 261538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10207.643480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10208.088822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10214.094410] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10215.279669] LustreError: 261895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10220.530464] LustreError: 261895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.580554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10223.588729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.277787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10230.939176] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10238.757498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10242.393910] Lustre: 263411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10247.137236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10247.150267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10247.173546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10247.177340] Lustre: Skipped 3 previous similar messages [10249.698318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10253.348539] Lustre: server umount lustre-MDT0000 complete [10254.818380] LustreError: 263178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10254.844768] LustreError: 263178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10256.952021] LustreError: 260377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957207 with bad export cookie 18331783006981822290 [10256.953994] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10256.975477] LustreError: 260377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10257.384320] Lustre: server umount lustre-MDT0001 complete [10270.443943] Lustre: server umount lustre-OST0000 complete [10284.798879] Lustre: server umount lustre-OST0001 complete [10304.267688] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:40:51 (1783957251) [10321.061568] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [10333.517653] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10372.990258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.311032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10373.385322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10373.518555] Lustre: lustre-MDT0000: new disk, initializing [10373.622592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10373.637984] Lustre: Skipped 1 previous similar message [10373.660599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10378.219664] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10389.943704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10390.152440] Lustre: 267667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10390.234629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10390.240959] Lustre: Skipped 1 previous similar message [10390.354979] Lustre: lustre-MDT0001: new disk, initializing [10390.548891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10390.579359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10396.136262] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10406.944794] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10414.046348] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10420.518298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10420.711476] Lustre: lustre-OST0000: new disk, initializing [10420.715569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10420.720785] Lustre: 269637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10422.711072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10422.719872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10422.827960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10425.769483] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10434.192406] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10438.827561] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10439.016977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10443.044232] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10443.236903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10448.354240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10448.362912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10448.379453] Lustre: Skipped 4 previous similar messages [10448.393124] Lustre: Skipped 4 previous similar messages [10458.598661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10458.604316] Lustre: Skipped 3 previous similar messages [10459.103315] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10459.281917] Lustre: server umount lustre-OST0000 complete [10467.299932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10467.311556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.883483] Lustre: server umount lustre-MDT0000 complete [10473.954341] LustreError: 268513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10473.978544] LustreError: 268513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10475.588718] LustreError: 267659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957426 with bad export cookie 18331783006981824390 [10475.594135] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10475.607417] LustreError: 267659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10476.092764] Lustre: server umount lustre-MDT0001 complete [10494.924164] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10506.047383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10506.430183] LustreError: 272570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.473411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10506.476591] Lustre: Skipped 2 previous similar messages [10511.492137] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10520.293490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10525.355657] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10528.609892] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10534.380557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10540.162439] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10548.022726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10548.223793] Lustre: lustre-OST0001: new disk, initializing [10548.227271] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10548.233116] Lustre: 274879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10549.420039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10549.433042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10549.581065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10554.207096] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10563.346829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10567.429815] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10573.562916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10573.831575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10582.506896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.509735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10582.528483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.528496] Lustre: Skipped 3 previous similar messages [10582.542653] Lustre: Skipped 3 previous similar messages [10588.199957] Lustre: server umount lustre-MDT0000 complete [10592.381734] LustreError: 274878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957543 with bad export cookie 18331783006981825272 [10592.397158] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10592.401798] LustreError: 274878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10592.736610] LustreError: 274075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10592.751509] LustreError: 274075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10592.823800] Lustre: server umount lustre-MDT0001 complete [10607.337360] Lustre: server umount lustre-OST0000 complete [10611.430418] Lustre: server umount lustre-OST0001 complete [10628.293591] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10638.573351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10639.324167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10639.332349] Lustre: Skipped 3 previous similar messages [10644.599724] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10653.998194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.286686] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10662.385175] Lustre: 279073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10670.565952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10678.639993] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10684.353364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10687.873475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10693.638246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10694.811106] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10703.070261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10706.850959] Lustre: 280952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10712.400803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10712.647745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10723.212960] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:47:51 (1783957671) [10739.699674] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.701962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10739.732518] Lustre: Skipped 5 previous similar messages [10739.737552] Lustre: Skipped 9 previous similar messages [10744.297935] Lustre: server umount lustre-MDT0000 complete [10744.812473] LustreError: 277934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.837298] LustreError: 277934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10747.639525] LustreError: 280954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957698 with bad export cookie 18331783006981831880 [10747.646304] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10747.656548] LustreError: 280954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10754.073722] Lustre: server umount lustre-MDT0001 complete [10763.659559] Lustre: server umount lustre-OST0000 complete [10766.811489] Lustre: server umount lustre-OST0001 complete [10773.177955] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [10782.984313] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10829.836796] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10838.071516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10838.254280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10838.278253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10838.367519] Lustre: lustre-MDT0000: new disk, initializing [10838.442922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10842.918744] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10851.866610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.989330] Lustre: 285192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10852.304501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10856.157823] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10867.211463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10867.604928] Lustre: 286822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10869.171996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10873.558114] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10882.735477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10882.965377] Lustre: 287694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10884.279353] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10888.776670] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10897.893478] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10901.391470] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10901.396790] Lustre: Skipped 1 previous similar message [10916.148541] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:51:04 (1783957864) [10918.880580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10918.889460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.902261] Lustre: Skipped 2 previous similar messages [10918.913508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10918.918817] Lustre: Skipped 5 previous similar messages [10920.424409] Lustre: server umount lustre-MDT0000 complete [10924.091301] LustreError: 285182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957874 with bad export cookie 18331783006981839545 [10924.095378] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10924.100423] LustreError: 285182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10924.661836] Lustre: server umount lustre-MDT0001 complete [10939.377595] Lustre: server umount lustre-OST0000 complete [10942.176076] Lustre: 250364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783957876/real 1783957876] req@ffff9442c1dc2a00 x1870614346716928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783957892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10943.517510] Lustre: server umount lustre-OST0001 complete [10952.239078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.899748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10952.902641] Lustre: Skipped 7 previous similar messages [10957.469094] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10964.978314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10969.138188] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10975.726646] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10981.056332] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10988.008822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10994.842646] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11001.611877] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11002.113767] LustreError: 291770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11002.130297] LustreError: 291770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [11016.212826] loop1: detected capacity change from 0 to 204800000 [11016.503797] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11016.519789] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11016.536765] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11020.994533] loop1: detected capacity change from 0 to 204800000 [11026.190129] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.277749] LustreError: 289927:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs18283: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11026.292084] LustreError: 289927:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs18283 [11026.300123] LustreError: 289927:0:(mgs_handler.c:517:mgs_target_reg()) fs18283-MDT0001: Error checking target index: rc = -13 [11026.313329] LustreError: fs18283-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11026.320557] LustreError: 293654:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11026.325558] LustreError: 293654:0:(tgt_mount.c:2129:server_put_super()) no obd fs18283-MDT0001 [11026.331108] LustreError: 293654:0:(tgt_mount.c:116:server_deregister_mount()) fs18283-MDT0001 not registered [11026.354638] Lustre: server umount fs18283-MDT0001 complete [11026.357936] LustreError: 293654:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11034.611630] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.757865] Lustre: Setting parameter fs18283-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18283-MDT0001 [11034.838304] Lustre: srv-fs18283-MDT0001: No data found on store. Initialize space. [11034.848291] Lustre: Skipped 4 previous similar messages [11034.937624] Lustre: fs18283-MDT0001: new disk, initializing [11034.942475] Lustre: Skipped 3 previous similar messages [11035.083766] Lustre: srv-fs18283-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11035.089673] Lustre: 294121:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18283-MDT0001-mdtlov: can't create new llog: rc = -115 [11035.100343] LustreError: 294121:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18283-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11039.270233] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11045.633980] Lustre: server umount fs18283-MDT0001 complete [11057.959931] loop1: detected capacity change from 0 to 204800000 [11063.133161] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.439797] Lustre: 295689:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18283-MDT0001-mdtlov: can't create new llog: rc = -115 [11063.449446] LustreError: 295689:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18283-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11067.294416] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11071.421619] Lustre: server umount fs18283-MDT0001 complete [11074.543726] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11083.238784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.257374] Lustre: Skipped 4 previous similar messages [11083.262056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11086.623642] Lustre: server umount lustre-MDT0000 complete [11095.235982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11095.357442] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11100.194747] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11101.170510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11108.130266] loop1: detected capacity change from 0 to 204800000 [11114.279911] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11114.568165] Lustre: 298137:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18283-MDT0001-mdtlov: can't create new llog: rc = -115 [11114.590195] LustreError: 298137:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18283-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11119.186425] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11124.028746] Lustre: server umount fs18283-MDT0001 complete [11146.722179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11146.731467] LustreError: Skipped 2 previous similar messages [11160.031120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11160.274441] Lustre: server umount lustre-OST0000 complete [11170.025943] Lustre: server umount lustre-MDT0000 complete [11173.523612] LustreError: 289920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958124 with bad export cookie 18331783006981842226 [11173.530571] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11173.532056] LustreError: 289920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11173.839623] Lustre: server umount lustre-MDT0001 complete [11183.050633] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [11185.596274] Key type lgssc unregistered [11185.805297] LNet: 300482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11185.817782] LNetError: 300482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11185.832604] LNet: Removed LNI 192.168.203.128@tcp [11186.611560] Key type .llcrypt unregistered [11186.613091] Key type ._llcrypt unregistered [11205.725764] Key type ._llcrypt registered [11205.728380] Key type .llcrypt registered [11205.838714] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11206.841926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11207.106745] alg: No test for adler32 (adler32-zlib) [11208.191762] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [11208.428649] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [11210.087179] Key type lgssc registered [11211.369532] Lustre: Echo OBD driver; http://www.lustre.org/ [11218.207252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11218.241938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.618663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11224.414620] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11230.184157] LustreError: 301756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.210119] LustreError: 301756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11232.925548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11233.518911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11238.273843] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11246.520375] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11252.697853] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11258.716836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11259.069474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11264.863508] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11265.192093] LustreError: 303584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.625821] LustreError: 303584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.435490] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11274.076661] LustreError: 303584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.899659] LustreError: 303584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.925453] LustreError: 303584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11289.072844] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11289.091776] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11293.089134] loop1: detected capacity change from 0 to 204800000 [11293.269205] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11293.281158] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11293.292884] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11297.816967] loop1: detected capacity change from 0 to 204800000 [11302.824352] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11302.945750] Lustre: fs1909-OST0000: new disk, initializing [11302.949817] Lustre: srv-fs1909-OST0000: No data found on store. Initialize space. [11302.957320] Lustre: 305492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11303.009582] Lustre: fs1909-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11305.165452] LustreError: 304718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.183697] LustreError: 304718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11308.054597] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11315.000789] Lustre: server umount fs1909-OST0000 complete [11324.899808] loop1: detected capacity change from 0 to 204800000 [11329.981936] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11330.182437] Lustre: fs1909-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.142918] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11340.270134] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11340.306153] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11345.404904] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11345.424591] loop2: detected capacity change from 0 to 204800000 [11345.608117] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11345.628946] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11345.637843] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11350.435395] loop2: detected capacity change from 0 to 204800000 [11356.200511] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.357477] LustreError: fs1909-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11356.370773] LustreError: fs1909-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11356.377630] LustreError: 308533:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11356.389849] LustreError: 308533:0:(tgt_mount.c:2129:server_put_super()) no obd fs1909-OST0001 [11356.394579] LustreError: 308533:0:(tgt_mount.c:116:server_deregister_mount()) fs1909-OST0001 not registered [11356.459244] Lustre: server umount fs1909-OST0001 complete [11356.464585] LustreError: 308533:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11360.749033] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11369.011691] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11369.149415] Lustre: fs1909-OST0001: new disk, initializing [11369.152782] Lustre: srv-fs1909-OST0001: No data found on store. Initialize space. [11369.158373] Lustre: 309086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11369.223667] Lustre: fs1909-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11374.620917] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11382.247036] Lustre: server umount fs1909-OST0000 complete [11390.464860] Lustre: server umount fs1909-OST0001 complete [11401.746828] loop1: detected capacity change from 0 to 204800000 [11406.832034] LustreError: 304947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.863763] LustreError: 304947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11406.907341] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11407.112464] Lustre: fs1909-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11413.454547] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11420.097503] loop2: detected capacity change from 0 to 204800000 [11425.979790] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11432.811767] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11442.997668] Lustre: server umount fs1909-OST0000 complete [11454.582128] Lustre: server umount fs1909-OST0001 complete [11468.257075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11468.282820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11468.291284] Lustre: Skipped 1 previous similar message [11473.107547] Lustre: server umount lustre-OST0000 complete [11478.501180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11478.508818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11478.535017] Lustre: Skipped 2 previous similar messages [11478.551951] Lustre: Skipped 1 previous similar message [11483.707123] Lustre: server umount lustre-MDT0000 complete [11487.364564] LustreError: 301735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958438 with bad export cookie 13098601339596567216 [11487.366515] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11487.376467] LustreError: 301735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11487.613769] Lustre: server umount lustre-MDT0001 complete [11499.132507] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [11501.994344] Key type lgssc unregistered [11502.313770] LNet: 315008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11502.319480] LNetError: 315008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11502.352175] LNet: Removed LNI 192.168.203.128@tcp [11503.215136] Key type .llcrypt unregistered [11503.217261] Key type ._llcrypt unregistered [11537.484722] Key type ._llcrypt registered [11537.486896] Key type .llcrypt registered [11537.612450] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [11552.607426] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11553.813172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11553.831859] alg: No test for adler32 (adler32-zlib) [11554.863972] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [11555.124996] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [11556.799349] Key type lgssc registered [11558.023160] Lustre: Echo OBD driver; http://www.lustre.org/ [11593.537386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11593.545704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11594.860684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11594.916860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11595.031279] Lustre: lustre-MDT0000: new disk, initializing [11595.142430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11595.178829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11600.032580] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11611.922969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11612.103218] Lustre: 318879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11612.174430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11612.181953] Lustre: Skipped 1 previous similar message [11612.292749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11612.315342] Lustre: lustre-MDT0001: new disk, initializing [11612.411693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11612.442776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11612.456755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11616.917762] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11625.862345] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11631.140430] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11637.339607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.609772] Lustre: lustre-OST0000: new disk, initializing [11637.614742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11637.622168] Lustre: 320849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11637.724775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11643.166441] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11645.504486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11645.511021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11645.616208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11650.874516] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11652.548604] Lustre: Mounted lustre-client [11653.425172] Lustre: Unmounted lustre-client [11662.822339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11662.824063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11662.830959] Lustre: Skipped 1 previous similar message [11667.889889] Lustre: server umount lustre-OST0000 complete [11673.057368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.075428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11673.082914] Lustre: Skipped 1 previous similar message [11677.785688] Lustre: server umount lustre-MDT0000 complete [11681.305460] LustreError: 318870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958632 with bad export cookie 798806569832842730 [11681.312740] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11681.316948] LustreError: 318870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11681.622913] Lustre: server umount lustre-MDT0001 complete [11692.293653] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [11694.711396] Key type lgssc unregistered [11711.840450] LNet: 1 local NIs in recovery (showing 1): 192.168.203.128@tcp [11712.415504] LNet: 323008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11712.430059] LNetError: 323008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11781.101646] LNet: Removed LNI 192.168.203.128@tcp [11782.167934] Key type .llcrypt unregistered [11782.169691] Key type ._llcrypt unregistered [11796.608139] Key type ._llcrypt registered [11796.609924] Key type .llcrypt registered [11796.669530] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [11805.673254] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11806.088885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11806.134048] alg: No test for adler32 (adler32-zlib) [11807.029471] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [11807.157157] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [11808.759325] Key type lgssc registered [11810.688204] Lustre: Echo OBD driver; http://www.lustre.org/ [11845.467536] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:06:33 (1783958793) [11852.358876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11852.373475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.681908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11853.730141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11853.835789] Lustre: lustre-MDT0000: new disk, initializing [11853.933517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11853.961940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11858.951542] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11870.846352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.055786] Lustre: 326204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11871.141640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11871.147752] Lustre: Skipped 1 previous similar message [11871.293858] Lustre: lustre-MDT0001: new disk, initializing [11871.415639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11871.467575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11871.479379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11876.317786] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11887.536812] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11894.252380] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11902.369099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11902.666838] Lustre: lustre-OST0000: new disk, initializing [11902.670878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11902.678834] Lustre: 328179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11902.805360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11909.243099] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11911.280783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11911.297632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11911.359211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11919.038490] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11937.251603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11937.256745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11937.274067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11937.297043] Lustre: Skipped 1 previous similar message [11940.210285] Lustre: server umount lustre-OST0000 complete [11947.490477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11947.504411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11947.507192] Lustre: Skipped 1 previous similar message [11947.537461] Lustre: Skipped 2 previous similar messages [11950.062298] Lustre: server umount lustre-MDT0000 complete [11954.144603] LustreError: 326197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958904 with bad export cookie 14505792126703525294 [11954.151248] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11954.165979] LustreError: 326197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11954.642812] Lustre: server umount lustre-MDT0001 complete [11966.208810] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [11969.440567] Key type lgssc unregistered [11969.694304] LNet: 330084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11969.705883] LNetError: 330084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11970.744729] LNet: Removed LNI 192.168.203.128@tcp [11971.759136] Key type .llcrypt unregistered [11971.760864] Key type ._llcrypt unregistered [11993.095561] Key type ._llcrypt registered [11993.097271] Key type .llcrypt registered [11993.203578] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11994.333190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11994.460326] alg: No test for adler32 (adler32-zlib) [11995.590516] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [11995.861251] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [11997.527180] Key type lgssc registered [11998.754828] Lustre: Echo OBD driver; http://www.lustre.org/ [12007.027317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12007.048048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12008.558650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12013.157992] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12019.173756] LustreError: 331363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12019.186700] LustreError: 331363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12022.339032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.636111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12027.176563] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12034.688527] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12040.705582] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12048.435564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12048.944298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12056.783329] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12065.091989] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12078.888559] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12080.885789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12087.525673] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [12087.786110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12087.817522] Lustre: Skipped 1 previous similar message [12092.197567] Lustre: server umount lustre-MDT0000 complete [12092.618648] LustreError: 331357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.268287] LustreError: 331344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959047 with bad export cookie 4552063639784426881 [12096.282270] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12096.283685] LustreError: 331344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12097.737764] LustreError: 332604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12097.751168] LustreError: 332604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12097.762455] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [12097.771507] Lustre: Skipped 3 previous similar messages [12098.020828] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12098.030101] Lustre: Skipped 1 previous similar message [12101.025110] LustreError: 331357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12101.025681] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12101.036710] LustreError: 331357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12101.055771] Lustre: Skipped 1 previous similar message [12102.689901] Lustre: server umount lustre-MDT0001 complete [12107.198289] Lustre: server umount lustre-OST0000 complete [12115.984323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12120.281687] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12123.069461] Lustre: 335545:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.128@tcp [12125.026896] Lustre: server umount MGS complete [12133.293404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12133.789405] LustreError: 336116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12133.897188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12138.258976] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12142.795708] LustreError: 336112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.817844] LustreError: 336112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12148.545481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.092364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12153.299755] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12162.355958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12162.853752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12168.899426] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12176.047292] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12177.511776] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12183.152455] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12185.598310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [12192.219785] Lustre: server umount lustre-OST0000 complete [12199.921892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12199.923138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.940625] Lustre: Skipped 1 previous similar message [12199.950524] Lustre: Skipped 1 previous similar message [12205.033989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12205.043351] Lustre: Skipped 1 previous similar message [12210.143278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12210.360995] Lustre: server umount lustre-MDT0000 complete [12211.167203] Lustre: 330673:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783959145/real 1783959145] req@0000000043529199 x1870616535824640/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783959161 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12211.199341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12211.212328] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.227184] LustreError: 336113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12214.075829] LustreError: 336096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959164 with bad export cookie 4552063639784427721 [12214.078303] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12214.085461] LustreError: 336096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12214.402224] Lustre: server umount lustre-MDT0001 complete [12225.077859] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12228.137618] Key type lgssc unregistered [12228.472582] LNet: 339554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12228.483271] LNetError: 339554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12229.543060] LNet: Removed LNI 192.168.203.128@tcp [12230.777265] Key type .llcrypt unregistered [12230.780892] Key type ._llcrypt unregistered [12251.232444] Key type ._llcrypt registered [12251.236902] Key type .llcrypt registered [12251.366951] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [12268.004689] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [12269.027406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12269.237192] alg: No test for adler32 (adler32-zlib) [12270.324277] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [12270.557456] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [12272.224216] Key type lgssc registered [12273.242919] Lustre: Echo OBD driver; http://www.lustre.org/ [12313.425643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12313.459959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.760439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12314.803528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12314.886873] Lustre: lustre-MDT0000: new disk, initializing [12314.954134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12314.975285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12319.977522] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12330.893805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12331.031604] Lustre: 342717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12331.080328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12331.084914] Lustre: Skipped 1 previous similar message [12331.149221] Lustre: lustre-MDT0001: new disk, initializing [12331.271200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12331.360982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12331.377500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12336.380641] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12346.181593] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12352.317946] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12359.533765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12359.948899] Lustre: lustre-OST0000: new disk, initializing [12359.954755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12359.982842] Lustre: 344689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12360.145304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12365.392812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12365.403934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12365.537168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12366.446978] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12376.509760] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12381.735325] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12381.905460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12387.052901] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12387.323469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12391.402881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12391.422973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12391.427900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12391.441193] Lustre: Skipped 1 previous similar message [12391.470968] Lustre: Skipped 1 previous similar message [12395.988069] Lustre: server umount lustre-OST0000 complete [12401.635331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12401.643400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12401.655064] Lustre: Skipped 1 previous similar message [12401.674233] Lustre: Skipped 1 previous similar message [12406.613994] Lustre: server umount lustre-MDT0000 complete [12410.853415] LustreError: 342710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959361 with bad export cookie 6339782761471677677 [12410.857948] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12410.867037] LustreError: 342710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12411.337241] Lustre: server umount lustre-MDT0001 complete [12421.742131] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:16:09 (1783959369) [12438.450832] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [12448.733422] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [12487.801050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12488.050386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12488.076197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12488.153280] Lustre: lustre-MDT0000: new disk, initializing [12488.251795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12488.277993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12492.842885] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12503.807519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12503.915389] Lustre: 349449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12503.988560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12503.996983] Lustre: Skipped 1 previous similar message [12504.090207] Lustre: lustre-MDT0001: new disk, initializing [12504.215785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12504.283106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12504.296187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12508.719179] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12519.030847] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12525.485964] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12533.069724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12533.322938] Lustre: lustre-OST0000: new disk, initializing [12533.331203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12533.341248] Lustre: 351415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12533.449421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12540.489570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12540.510279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12540.647650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12541.178537] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12551.402481] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12586.986125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12586.999271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12587.000015] Lustre: Skipped 1 previous similar message [12587.001341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12587.015354] LustreError: Skipped 1 previous similar message [12587.032986] Lustre: Skipped 1 previous similar message [12592.099974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12592.131481] Lustre: Skipped 1 previous similar message [12592.738245] Lustre: server umount lustre-OST0000 complete [12596.625215] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12596.630467] LustreError: Skipped 1 previous similar message [12596.640838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12596.655789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12603.094038] Lustre: server umount lustre-MDT0000 complete [12607.063374] LustreError: 351425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959557 with bad export cookie 6339782761471678559 [12607.064427] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12607.073355] LustreError: 351425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12607.404534] Lustre: server umount lustre-MDT0001 complete [12619.129258] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12622.237228] Key type lgssc unregistered [12622.488717] LNet: 353590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12622.501242] LNetError: 353590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12623.545835] LNet: Removed LNI 192.168.203.128@tcp [12624.540148] Key type .llcrypt unregistered [12624.543755] Key type ._llcrypt unregistered [12726.136807] Key type ._llcrypt registered [12726.140365] Key type .llcrypt registered [12726.319959] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12726.677603] Key type .llcrypt unregistered [12726.683578] Key type ._llcrypt unregistered [12809.642400] Key type ._llcrypt registered [12809.648031] Key type .llcrypt registered [12809.787188] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12810.118487] Key type .llcrypt unregistered [12810.131555] Key type ._llcrypt unregistered [12842.740361] Key type ._llcrypt registered [12842.744552] Key type .llcrypt registered [12842.949212] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12843.347403] Key type .llcrypt unregistered [12843.353897] Key type ._llcrypt unregistered [12874.084129] Key type ._llcrypt registered [12874.086106] Key type .llcrypt registered [12874.239611] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12874.619443] Key type .llcrypt unregistered [12874.622150] Key type ._llcrypt unregistered [12957.313988] Key type ._llcrypt registered [12957.316300] Key type .llcrypt registered [12957.404154] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12957.637489] Key type .llcrypt unregistered [12957.641403] Key type ._llcrypt unregistered [12988.430990] Key type ._llcrypt registered [12988.434168] Key type .llcrypt registered [12988.530662] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12988.827456] Key type .llcrypt unregistered [12988.829499] Key type ._llcrypt unregistered [13021.676442] Key type ._llcrypt registered [13021.681332] Key type .llcrypt registered [13021.908925] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13022.311590] Key type .llcrypt unregistered [13022.313656] Key type ._llcrypt unregistered [13060.909789] Key type ._llcrypt registered [13060.912547] Key type .llcrypt registered [13061.052405] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13061.516834] Key type .llcrypt unregistered [13061.520332] Key type ._llcrypt unregistered [13096.593062] Key type ._llcrypt registered [13096.596663] Key type .llcrypt registered [13096.735380] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13097.106522] Key type .llcrypt unregistered [13097.111686] Key type ._llcrypt unregistered [13129.547544] Key type ._llcrypt registered [13129.553384] Key type .llcrypt registered [13129.707829] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13130.096942] Key type .llcrypt unregistered [13130.105130] Key type ._llcrypt unregistered [13195.600471] Key type ._llcrypt registered [13195.601863] Key type .llcrypt registered [13195.705889] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13195.951812] Key type .llcrypt unregistered [13195.965648] Key type ._llcrypt unregistered [13229.132357] Key type ._llcrypt registered [13229.133971] Key type .llcrypt registered [13229.238857] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13229.527856] Key type .llcrypt unregistered [13229.530574] Key type ._llcrypt unregistered [13263.188520] Key type ._llcrypt registered [13263.191220] Key type .llcrypt registered [13263.289390] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13263.693682] Key type .llcrypt unregistered [13263.701109] Key type ._llcrypt unregistered [13318.069601] Key type ._llcrypt registered [13318.076937] Key type .llcrypt registered [13318.185745] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [13333.803961] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [13334.697814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13334.756607] alg: No test for adler32 (adler32-zlib) [13335.803549] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [13336.032153] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [13337.687157] Key type lgssc registered [13338.845032] Lustre: Echo OBD driver; http://www.lustre.org/ [13376.419418] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13376.464973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13377.726227] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13377.771973] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13377.886387] Lustre: tst250fs-MDT0000: new disk, initializing [13378.012409] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13378.046382] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13384.260692] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13398.744527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13398.853611] Lustre: 373911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13398.914704] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13398.923297] Lustre: Skipped 1 previous similar message [13399.004600] Lustre: tst250fs-MDT0001: new disk, initializing [13399.188818] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13399.239373] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13399.250043] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13405.380943] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13411.361551] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13419.065179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13419.437491] Lustre: tst250fs-OST0000: new disk, initializing [13419.445749] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13419.453940] Lustre: 375546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13419.625306] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13424.716040] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13424.731613] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13424.880480] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13426.590946] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13438.426364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13438.673101] Lustre: tst250fs-OST0001: new disk, initializing [13438.680694] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13438.689310] Lustre: 376418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13438.824445] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13444.749779] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13444.761786] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13444.896507] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13445.822301] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13451.217688] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13454.041781] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13454.054732] Lustre: Skipped 2 previous similar messages [13475.823258] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13475.827553] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13475.837456] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13475.854291] Lustre: Skipped 1 previous similar message [13475.903647] Lustre: Skipped 3 previous similar messages [13478.374650] Lustre: server umount tst250fs-MDT0000 complete [13483.376153] LustreError: 373905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783960434 with bad export cookie 12504117512353267331 [13483.406121] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13483.864403] Lustre: server umount tst250fs-MDT0001 complete [13497.488238] Lustre: server umount tst250fs-OST0000 complete [13511.364708] Lustre: server umount tst250fs-OST0001 complete [13528.599559] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [13536.119829] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [13564.227266] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:35:12 (1783960512) [13565.675635] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13582.288994] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [13591.898352] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [13623.024174] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13284 sec ========= 12:36:10 (1783960570) [13624.747345] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:36:12 (1783960572) === [13627.974504] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:36:15 (1783960575) === [13670.581219] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13673.302267] Key type lgssc unregistered [13673.652409] LNet: 385563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13673.674216] LNetError: 385563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13674.727908] LNet: Removed LNI 192.168.203.128@tcp [13675.667224] Key type .llcrypt unregistered [13675.669345] Key type ._llcrypt unregistered