[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 664214734 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003127] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007906] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011072] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.021727] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026093] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031014] *** VALIDATE tmpfs *** [ 0.033033] *** VALIDATE proc *** [ 0.034274] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036306] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042875] debug: unmapping init [mem 0xffffffffb1659000-0xffffffffb1660fff] [ 0.045886] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046830] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047039] ... version: 2 [ 0.048019] ... bit width: 48 [ 0.049018] ... generic registers: 4 [ 0.050020] ... value mask: 0000ffffffffffff [ 0.051020] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054483] rcu: Hierarchical SRCU implementation. [ 0.056799] smp: Bringing up secondary CPUs ... [ 0.057633] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.062548] smp: Brought up 1 node, 4 CPUs [ 0.064024] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.274300] node 0 deferred pages initialised in 206ms [ 0.278304] devtmpfs: initialized [ 0.279255] x86/mm: Memory block size: 128MB [ 0.281000] gcov: version magic: 0x41383552 [ 0.285409] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.286075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.289321] pinctrl core: initialized pinctrl subsystem [ 0.291235] [ 0.291879] ************************************************************* [ 0.295015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297011] ** ** [ 0.300015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.303015] ** ** [ 0.306014] ** This means that this kernel is built to expose internal ** [ 0.308009] ** IOMMU data structures, which may compromise security on ** [ 0.310013] ** your system. ** [ 0.312016] ** ** [ 0.315015] ** If you see this message and you are not debugging the ** [ 0.318021] ** kernel, report this immediately to your vendor! ** [ 0.320013] ** ** [ 0.322014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.325017] ************************************************************* [ 0.328260] NET: Registered protocol family 16 [ 0.330496] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.333078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.336103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.340041] cpuidle: using governor menu [ 0.343032] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.345577] PCI: Using configuration type 1 for base access [ 0.348132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.359094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.361032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.365037] cryptd: max_cpu_qlen set to 1000 [ 0.367269] ACPI: Added _OSI(Module Device) [ 0.369021] ACPI: Added _OSI(Processor Device) [ 0.371019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.372015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.377516] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.385624] ACPI: Interpreter enabled [ 0.386078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.388035] ACPI: Using IOAPIC for interrupt routing [ 0.390139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.394403] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.404069] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.406044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.409032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.412095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.418227] acpiphp: Slot [2] registered [ 0.419183] acpiphp: Slot [5] registered [ 0.421128] acpiphp: Slot [6] registered [ 0.422119] acpiphp: Slot [7] registered [ 0.424148] acpiphp: Slot [8] registered [ 0.425130] acpiphp: Slot [9] registered [ 0.427156] acpiphp: Slot [10] registered [ 0.428185] acpiphp: Slot [3] registered [ 0.430120] acpiphp: Slot [4] registered [ 0.432147] acpiphp: Slot [11] registered [ 0.433100] acpiphp: Slot [12] registered [ 0.435123] acpiphp: Slot [13] registered [ 0.436245] acpiphp: Slot [14] registered [ 0.438110] acpiphp: Slot [15] registered [ 0.440168] acpiphp: Slot [16] registered [ 0.441116] acpiphp: Slot [17] registered [ 0.443108] acpiphp: Slot [18] registered [ 0.444129] acpiphp: Slot [19] registered [ 0.446119] acpiphp: Slot [20] registered [ 0.448140] acpiphp: Slot [21] registered [ 0.449085] acpiphp: Slot [22] registered [ 0.450161] acpiphp: Slot [23] registered [ 0.452098] acpiphp: Slot [24] registered [ 0.454138] acpiphp: Slot [25] registered [ 0.455123] acpiphp: Slot [26] registered [ 0.457121] acpiphp: Slot [27] registered [ 0.458062] acpiphp: Slot [28] registered [ 0.459143] acpiphp: Slot [29] registered [ 0.460096] acpiphp: Slot [30] registered [ 0.461114] acpiphp: Slot [31] registered [ 0.464450] PCI host bridge to bus 0000:00 [ 0.466029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.469027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.471023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.474028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.477027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.480038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.482680] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.485181] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.489479] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.499674] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.505019] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.508029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.510022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.513041] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.516555] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.520073] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.523042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.527074] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.534022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.548017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.554024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.560000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.567017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.578017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.603020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.614698] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.624019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.631025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.656017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.667589] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.674018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.680017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.697020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.707219] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.717016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.724019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.746017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.758580] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.771018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.778017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.800018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.814192] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.823016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.831015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.852017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.862622] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.865771] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.869475] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.872470] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.876259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.881103] iommu: Default domain type: Passthrough [ 0.882747] SCSI subsystem initialized [ 0.885202] ACPI: bus type USB registered [ 0.887402] usbcore: registered new interface driver usbfs [ 0.889180] usbcore: registered new interface driver hub [ 0.892097] usbcore: registered new device driver usb [ 0.895188] pps_core: LinuxPPS API ver. 1 registered [ 0.897011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.900064] PTP clock support registered [ 0.904005] EDAC MC: Ver: 3.0.0 [ 0.906140] PCI: Using ACPI for IRQ routing [ 0.907835] NetLabel: Initializing [ 0.911019] NetLabel: domain hash size = 128 [ 0.913008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.915080] NetLabel: unlabeled traffic allowed by default [ 0.918048] vgaarb: loaded [ 0.919305] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.922018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.929000] clocksource: Switched to clocksource kvm-clock [ 1.042790] VFS: Disk quotas dquot_6.6.0 [ 1.044567] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.047179] *** VALIDATE ramfs *** [ 1.048747] *** VALIDATE hugetlbfs *** [ 1.050391] pnp: PnP ACPI init [ 1.053301] pnp: PnP ACPI: found 6 devices [ 1.086140] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.089909] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.092600] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.095051] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.097748] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.100326] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.103418] NET: Registered protocol family 2 [ 1.106086] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.111214] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.114884] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.120453] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.124516] TCP: Hash tables configured (established 65536 bind 65536) [ 1.128762] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.132314] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.135481] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.138103] NET: Registered protocol family 1 [ 1.141518] RPC: Registered named UNIX socket transport module. [ 1.144098] RPC: Registered udp transport module. [ 1.145960] RPC: Registered tcp transport module. [ 1.147916] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.149759] NET: Registered protocol family 44 [ 1.151478] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.153565] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.155552] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.158104] PCI: CLS 0 bytes, default 64 [ 1.160107] Unpacking initramfs... [ 2.562588] debug: unmapping init [mem 0xffff992a7cc54000-0xffff992a7ffbffff] [ 2.567028] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.569589] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.572684] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.083810] Initialise system trusted keyrings [ 3.088100] Key type blacklist registered [ 3.090307] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.099795] zbud: loaded [ 3.103129] *** VALIDATE nfs *** [ 3.104737] *** VALIDATE nfs4 *** [ 3.106513] pstore: using deflate compression [ 3.110304] Platform Keyring initialized [ 3.224423] NET: Registered protocol family 38 [ 3.226542] Key type asymmetric registered [ 3.228256] Asymmetric key parser 'x509' registered [ 3.230282] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.233737] io scheduler mq-deadline registered [ 3.235473] io scheduler kyber registered [ 3.237302] io scheduler bfq registered [ 3.239183] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.242444] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.245652] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.249087] ACPI: Power Button [PWRF] [ 3.255274] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.262895] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.280813] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.289208] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.309353] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.341387] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.370077] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.374901] Non-volatile memory driver v1.3 [ 3.376865] Linux agpgart interface v0.103 [ 3.408186] virtio_blk virtio1: [vda] 150008 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.411228] vda: detected capacity change from 0 to 76804096 [ 3.426060] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.429220] vdb: detected capacity change from 0 to 1073741824 [ 3.443710] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.446715] vdc: detected capacity change from 0 to 2621440000 [ 3.465619] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.468433] vdd: detected capacity change from 0 to 2621440000 [ 3.487734] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.490896] vde: detected capacity change from 0 to 4294967296 [ 3.507754] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.511564] vdf: detected capacity change from 0 to 4294967296 [ 3.520428] libphy: Fixed MDIO Bus: probed [ 3.529427] usbcore: registered new interface driver usbserial_generic [ 3.532109] usbserial: USB Serial support registered for generic [ 3.534565] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.538367] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.540400] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.542867] mousedev: PS/2 mouse device common for all mice [ 3.546184] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.546556] rtc_cmos 00:05: RTC can wake from S4 [ 3.554202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.554519] rtc_cmos 00:05: registered as rtc0 [ 3.559362] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.563080] intel_pstate: CPU model not supported [ 3.566263] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.570358] hid: raw HID events driver (C) Jiri Kosina [ 3.573163] usbcore: registered new interface driver usbhid [ 3.575460] usbhid: USB HID core driver [ 3.577029] drop_monitor: Initializing network drop monitor service [ 3.580092] Initializing XFRM netlink socket [ 3.582596] NET: Registered protocol family 10 [ 3.585826] Segment Routing with IPv6 [ 3.587426] NET: Registered protocol family 17 [ 3.590029] mpls_gso: MPLS GSO support [ 3.596213] RAS: Correctable Errors collector initialized. [ 3.598708] AVX version of gcm_enc/dec engaged. [ 3.601699] AES CTR mode by8 optimization enabled [ 3.684358] sched_clock: Marking stable (3684337246, 0)->(4630343761, -946006515) [ 3.688463] registered taskstats version 1 [ 3.691132] Loading compiled-in X.509 certificates [ 3.693414] zswap: loaded using pool lzo/zbud [ 3.718077] Key type big_key registered [ 3.732615] Key type encrypted registered [ 3.734309] ima: No TPM chip found, activating TPM-bypass! [ 3.736573] ima: Allocated hash algorithm: sha1 [ 3.738340] ima: No architecture policies found [ 3.739706] evm: Initialising EVM extended attributes: [ 3.741281] evm: security.selinux [ 3.742722] evm: security.ima [ 3.743830] evm: security.capability [ 3.745191] evm: HMAC attrs: 0x1 [ 3.747818] rtc_cmos 00:05: setting system clock to 2026-09-15 16:13:37 UTC (1789488817) [ 3.755146] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 3.758768] debug: unmapping init [mem 0xffffffffb1382000-0xffffffffb1658fff] [ 3.769245] Write protecting the kernel read-only data: 28672k [ 3.773277] debug: unmapping init [mem 0xffffffffafa03000-0xffffffffafbfffff] [ 3.775495] debug: unmapping init [mem 0xffffffffb0314000-0xffffffffb03fffff] [ 3.808802] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.817943] systemd[1]: Detected virtualization kvm. [ 3.819555] systemd[1]: Detected architecture x86-64. [ 3.821203] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.847168] systemd[1]: No hostname configured. [ 3.849634] systemd[1]: Set hostname to . [ 3.851864] random: systemd: uninitialized urandom read (16 bytes read) [ 3.854741] systemd[1]: Initializing machine ID from random generator. [ 3.981499] random: systemd: uninitialized urandom read (16 bytes read) [ 3.985241] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.990635] random: systemd: uninitialized urandom read (16 bytes read) [ 3.993363] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.997491] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.604434] device-mapper: uevent: version 1.0.3 [ 4.606350] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut p[ 4.958176] random: fast init done re-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.351294] virtio_net virtio0 ens2: renamed from eth0 [ 5.415326] scsi host0: ata_piix [ 5.453579] scsi host1: ata_piix [ 5.455408] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.458179] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.875471] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.255259] random: crng init done [ 10.257745] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.168619] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.187948] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.094844] SELinux: Disabled at runtime. [ 16.254839] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.273487] systemd[1]: Detected virtualization kvm. [ 16.278168] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.173350] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.185667] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.208928] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.222523] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.233941] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.269921] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.282590] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [FAILED[ 18.462398] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 19.709695] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.866608] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.005717] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.504476] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.609430] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.897573] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 27.478059] NFS: Registering the id_resolver key type [ 27.487555] Key type id_resolver registered [ 27.492595] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 43.624890] hrtimer: interrupt took 3007012 ns [ 99.620791] libcfs: loading out-of-tree module taints kernel. [ 99.796150] Key type ._llcrypt registered [ 99.801867] Key type .llcrypt registered [ 99.992245] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 119.479315] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 121.635924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.652031] alg: No test for adler32 (adler32-zlib) [ 123.228435] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 124.193741] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 126.079276] Key type lgssc registered [ 128.546824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.325108] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.953381] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 201.820930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.851906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.100702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.140783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.274526] Lustre: lustre-MDT0000: new disk, initializing [ 203.403200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.426750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.243725] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.025910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.198058] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.236766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.242119] Lustre: Skipped 1 previous similar message [ 222.330673] Lustre: lustre-MDT0001: new disk, initializing [ 222.422431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.446969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 222.454714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.827990] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.600972] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.256232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.569565] Lustre: lustre-OST0000: new disk, initializing [ 244.578295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 244.590570] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.667444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.461332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 250.482482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.591010] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.468102] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.180917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.329095] Lustre: lustre-OST0001: new disk, initializing [ 271.337973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.350928] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.469777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.709768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.744724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.756957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.827937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 293.941938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 302.164398] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 310.279962] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 316.906800] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 322.454703] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 325.187960] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 328.113470] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 329.730709] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 12:19:01 EDT 2026 [ 349.322480] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 351.054339] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 369.120302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.122478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.143220] Lustre: Skipped 2 previous similar messages [ 369.157829] Lustre: Skipped 3 previous similar messages [ 373.293942] Lustre: server umount lustre-MDT0000 complete [ 377.046609] LustreError: 9530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489190 with bad export cookie 2132304206921452101 [ 377.048210] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.055624] LustreError: 9530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 379.369803] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.370172] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.386342] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.402540] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 379.432180] Lustre: Skipped 2 previous similar messages [ 382.432186] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.432372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.454639] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 382.481498] Lustre: Skipped 2 previous similar messages [ 383.621846] Lustre: server umount lustre-MDT0001 complete [ 387.739242] Lustre: server umount lustre-OST0000 complete [ 392.151818] Lustre: server umount lustre-OST0001 complete [ 398.936102] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 408.584272] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 445.304156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.590823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.619745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 445.754483] Lustre: lustre-MDT0000: new disk, initializing [ 445.837411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.859628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.699118] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 462.346750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.508497] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.561339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.566242] Lustre: Skipped 1 previous similar message [ 462.641366] Lustre: lustre-MDT0001: new disk, initializing [ 462.727465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 462.779116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 462.805937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.608211] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 481.823066] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 489.378464] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.752234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.095639] Lustre: lustre-OST0000: new disk, initializing [ 497.101427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 497.108205] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 497.283178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.354704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.367693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.536983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.263647] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 514.166699] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.118925] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.329555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 524.106186] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 524.324983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 529.380570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.392160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.393243] Lustre: Skipped 1 previous similar message [ 529.403787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.497858] Lustre: server umount lustre-OST0000 complete [ 539.616509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.623701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.626857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.626866] Lustre: Skipped 1 previous similar message [ 539.655516] Lustre: Skipped 1 previous similar message [ 542.526087] Lustre: server umount lustre-MDT0000 complete [ 544.737125] LustreError: 16314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.759245] LustreError: 16314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 546.233793] LustreError: 15883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489359 with bad export cookie 2132304206921453634 [ 546.241471] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.245422] LustreError: 15883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 546.511786] Lustre: server umount lustre-MDT0001 complete [ 551.745663] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:22:43 (1789489363) [ 569.438812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.860275] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.948306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.876314] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 574.948182] LustreError: 19931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.737715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.857793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 596.735163] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 604.295727] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 606.772821] Lustre: Failing over lustre-MDT0000 [ 607.325974] Lustre: server umount lustre-MDT0000 complete [ 607.723124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 607.741485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.766584] LustreError: 19931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.785023] LustreError: 19931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 627.168590] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.194906] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 627.658046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.831032] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 628.139795] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 632.744852] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 633.323601] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 633.331323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 633.367396] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 642.786459] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 647.655438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.660042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.673564] Lustre: Skipped 2 previous similar messages [ 647.685515] Lustre: Skipped 2 previous similar messages [ 653.435688] Lustre: server umount lustre-MDT0000 complete [ 658.246550] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489471 with bad export cookie 2132304206921455048 [ 658.260162] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.275213] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 658.773689] Lustre: server umount lustre-MDT0001 complete [ 669.476895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.207811] LustreError: 23266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.240443] LustreError: 23266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 670.361264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.367369] Lustre: Skipped 2 previous similar messages [ 675.699631] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 685.473778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.851970] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 698.710345] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 704.654177] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 706.652398] Lustre: Failing over lustre-MDT0000 [ 706.866693] Lustre: server umount lustre-MDT0000 complete [ 710.637526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.646032] Lustre: Skipped 1 previous similar message [ 725.453381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.613234] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.838946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 731.048657] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 731.107065] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 731.121513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 731.130142] Lustre: Skipped 1 previous similar message [ 731.145062] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 741.333162] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 745.201498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.218802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 745.226661] Lustre: Skipped 3 previous similar messages [ 751.539495] Lustre: server umount lustre-MDT0000 complete [ 755.599158] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489569 with bad export cookie 2132304206921455986 [ 755.603228] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.609345] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 755.984519] Lustre: server umount lustre-MDT0001 complete [ 766.157809] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:26:17 (1789489577) [ 785.128432] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 798.123704] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 840.333516] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 849.578985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.780858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 849.818681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 849.907553] Lustre: lustre-MDT0000: new disk, initializing [ 849.985882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.996584] Lustre: Skipped 2 previous similar messages [ 850.012362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 854.086818] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 864.684716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.816746] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 865.054513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 869.387607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 878.250888] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 883.518160] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 890.564995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 890.874125] Lustre: 31610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 892.671705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 892.717957] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 892.789561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 892.812522] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 897.108518] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 907.508723] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.454545] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 938.978110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 938.978469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.987257] LustreError: Skipped 1 previous similar message [ 938.991610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 939.009042] Lustre: Skipped 2 previous similar messages [ 939.035825] Lustre: Skipped 3 previous similar messages [ 948.191754] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 948.503831] Lustre: server umount lustre-OST0000 complete [ 958.384574] Lustre: server umount lustre-MDT0000 complete [ 962.532237] LustreError: 30506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.576021] LustreError: 30506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 963.644157] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489777 with bad export cookie 2132304206921456385 [ 963.656727] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 963.661160] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 964.151684] Lustre: server umount lustre-MDT0001 complete [ 975.744370] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 978.670207] Key type lgssc unregistered [ 978.910596] LNet: 33587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.920662] LNetError: 33587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.940995] LNet: Removed LNI 192.168.204.126@tcp [ 979.862274] Key type .llcrypt unregistered [ 979.867684] Key type ._llcrypt unregistered [ 1007.687966] Key type ._llcrypt registered [ 1007.696900] Key type .llcrypt registered [ 1007.874828] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1009.823095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1010.019737] alg: No test for adler32 (adler32-zlib) [ 1011.102723] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1011.384692] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1013.051115] Key type lgssc registered [ 1014.639217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1027.674363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1027.735412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.792889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.066151] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1039.335025] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.350567] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1044.458858] LustreError: 35003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.705502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.297647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1051.461594] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1059.089795] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1066.352619] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1074.198655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1074.798570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.599378] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1089.108093] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1091.194803] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1103.992420] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:31:55 (1789489915) [ 1106.017635] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1107.827385] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1115.556170] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:32:06 (1789489926) [ 1126.232532] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:32:16 (1789489936) [ 1127.178377] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1138.017261] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1138.024806] Lustre: Skipped 2 previous similar messages [ 1140.267810] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:32:31 (1789489951) [ 1142.804223] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1142.808940] Lustre: Skipped 2 previous similar messages [ 1152.203285] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1152.212728] Lustre: Skipped 4 previous similar messages [ 1167.908913] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:32:58 (1789489978) [ 1170.345451] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1174.322936] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.078852] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:33:15 (1789489995) [ 1186.503357] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1186.510169] Lustre: Skipped 6 previous similar messages [ 1202.248203] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:33:32 (1789490012) [ 1224.673348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.673979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1224.697124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1224.713366] Lustre: Skipped 1 previous similar message [ 1226.014366] Lustre: server umount lustre-OST0000 complete [ 1229.796763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.803457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.809276] Lustre: Skipped 2 previous similar messages [ 1229.819167] Lustre: Skipped 1 previous similar message [ 1234.915255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.924220] Lustre: Skipped 1 previous similar message [ 1240.033778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1240.039235] Lustre: Skipped 1 previous similar message [ 1244.128442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1244.332022] Lustre: server umount lustre-MDT0000 complete [ 1245.155759] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.172767] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1247.465232] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490061 with bad export cookie 17646913827411638675 [ 1247.469513] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.484505] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1247.861315] Lustre: server umount lustre-MDT0001 complete [ 1262.969255] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1267.739731] Key type lgssc unregistered [ 1268.239335] LNet: 42863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.248666] LNetError: 42863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.291486] LNet: Removed LNI 192.168.204.126@tcp [ 1270.753503] Key type .llcrypt unregistered [ 1270.755577] Key type ._llcrypt unregistered [ 1290.556638] Key type ._llcrypt registered [ 1290.559454] Key type .llcrypt registered [ 1290.669956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 1307.373844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1308.591916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.653888] alg: No test for adler32 (adler32-zlib) [ 1309.925492] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1310.224371] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1311.912693] Key type lgssc registered [ 1313.178713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1353.629272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1353.664053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.037664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1355.100685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1355.218949] Lustre: lustre-MDT0000: new disk, initializing [ 1355.307098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.348828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1360.324294] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1371.691962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.871123] Lustre: 46014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1371.958856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1371.963147] Lustre: Skipped 1 previous similar message [ 1372.056387] Lustre: lustre-MDT0001: new disk, initializing [ 1372.190588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.220198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1372.233455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1377.807375] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1388.402616] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1394.980723] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1403.138804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.426298] Lustre: lustre-OST0000: new disk, initializing [ 1403.441376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1403.452766] Lustre: 47985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1403.580243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.127798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1409.141925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1409.244193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1410.186134] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1420.525679] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1427.029564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1427.234605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1432.785343] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1433.057794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1440.240723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.246239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1440.265197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.639637] Lustre: server umount lustre-OST0000 complete [ 1448.929042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1448.951478] LustreError: Skipped 1 previous similar message [ 1448.953511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.993713] Lustre: Skipped 1 previous similar message [ 1449.000463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.027076] Lustre: Skipped 1 previous similar message [ 1450.464348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.468147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.535100] Lustre: Skipped 1 previous similar message [ 1453.704068] Lustre: server umount lustre-MDT0000 complete [ 1458.215219] LustreError: 46820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490271 with bad export cookie 18400095482332764494 [ 1458.224473] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1458.251220] LustreError: 46820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1458.803768] Lustre: server umount lustre-MDT0001 complete [ 1469.433936] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:38:00 (1789490280) [ 1476.351654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.706484] LustreError: 50446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1476.724445] LustreError: 50446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1476.800152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.642455] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1482.218050] LustreError: 50447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.322729] LustreError: 50446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.154992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.433628] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1491.646845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1496.363480] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1504.469664] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1510.425601] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1517.397496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1517.915918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.869699] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1533.797610] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1541.035417] LustreError: 52974:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1542.693575] Lustre: Modifying parameter general.timeout=1 in log params [ 1544.190598] Lustre: Modifying parameter general.timeout=2 in log params [ 1546.902805] Lustre: Modifying parameter general.timeout=4 in log params [ 1546.911787] Lustre: Skipped 1 previous similar message [ 1551.221656] Lustre: Modifying parameter general.timeout=7 in log params [ 1551.232399] Lustre: Skipped 2 previous similar messages [ 1559.298394] Lustre: Modifying parameter general.timeout=14 in log params [ 1559.307645] Lustre: Skipped 6 previous similar messages [ 1575.941380] Lustre: Modifying parameter general.timeout=30 in log params [ 1575.951299] Lustre: Skipped 15 previous similar messages [ 1608.411902] Lustre: Modifying parameter general.timeout=68 in log params [ 1608.415252] Lustre: Skipped 37 previous similar messages [ 1675.531526] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:41:26 (1789490486) [ 1676.593727] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1676.597703] Lustre: Skipped 62 previous similar messages [ 1678.522360] Lustre: 59925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1680.226365] Lustre: 60023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1684.114832] Lustre: 60238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1684.134628] Lustre: 60238:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1688.185948] Lustre: 60444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1688.194274] Lustre: 60444:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1718.967765] Lustre: 61801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1719.024630] LustreError: 61801:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1725.574138] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:42:15 (1789490535) [ 1743.842229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.846679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.869822] Lustre: Skipped 2 previous similar messages [ 1743.889558] Lustre: Skipped 2 previous similar messages [ 1749.341986] Lustre: server umount lustre-MDT0000 complete [ 1753.730532] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490567 with bad export cookie 18400095482332765376 [ 1753.779154] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1753.799438] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.127406] LustreError: 50442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1754.132169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.704096] Lustre: server umount lustre-MDT0001 complete [ 1761.758340] Lustre: server umount lustre-OST0000 complete [ 1792.437774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.057535] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1793.542574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.455740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1811.490038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.776974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.876123] Lustre: 64334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1820.080361] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1828.497298] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1835.271682] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1844.087535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1844.410137] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1844.846295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.854443] Lustre: Skipped 1 previous similar message [ 1851.847744] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1860.495305] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1862.206442] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1862.214662] Lustre: Skipped 22 previous similar messages [ 1862.267122] Lustre: 66902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1871.840617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1871.852466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.879681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1871.889894] Lustre: Skipped 3 previous similar messages [ 1876.405908] Lustre: server umount lustre-OST0000 complete [ 1892.832921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.835239] Lustre: Skipped 6 previous similar messages [ 1895.402302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1895.659182] Lustre: server umount lustre-MDT0000 complete [ 1897.956446] LustreError: 65405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.981333] LustreError: 65405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1900.826321] LustreError: 64326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490714 with bad export cookie 18400095482332768540 [ 1900.834616] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.843756] LustreError: 64326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1901.504460] Lustre: server umount lustre-MDT0001 complete [ 1916.113740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1919.715820] Key type lgssc unregistered [ 1920.085612] LNet: 68116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.100238] LNetError: 68116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.116976] LNet: Removed LNI 192.168.204.126@tcp [ 1920.887063] Key type .llcrypt unregistered [ 1920.896607] Key type ._llcrypt unregistered [ 1946.327612] Key type ._llcrypt registered [ 1946.332538] Key type .llcrypt registered [ 1946.533895] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1947.613736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1947.655091] alg: No test for adler32 (adler32-zlib) [ 1948.695909] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1948.993407] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1950.663210] Key type lgssc registered [ 1951.738344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1960.095418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1960.147214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.908257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.054763] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1972.207968] LustreError: 69527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.245346] LustreError: 69527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1977.313317] LustreError: 69528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.901840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.789535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.289324] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1993.534529] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1999.806541] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2006.507898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2007.182103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.133310] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2023.550036] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2042.269408] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2052.590160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.601719] Lustre: Skipped 1 previous similar message [ 2052.603581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.609436] Lustre: Skipped 2 previous similar messages [ 2057.698797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.703867] Lustre: Skipped 1 previous similar message [ 2058.136518] Lustre: server umount lustre-MDT0000 complete [ 2062.329693] LustreError: 69507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490876 with bad export cookie 2124480278577322057 [ 2062.338528] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.817514] Lustre: server umount lustre-MDT0001 complete [ 2076.490026] Lustre: server umount lustre-OST0000 complete [ 2108.220506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.693737] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2108.754419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2109.147922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.390703] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2123.840790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.944556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2124.001239] Lustre: 74420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2124.446052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.458483] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2137.214716] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2144.091047] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2151.591506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2151.773951] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2152.125280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.385642] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2169.116816] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2171.808397] Lustre: 76995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2174.749867] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2181.095457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2181.099323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.118339] LustreError: Skipped 1 previous similar message [ 2181.126361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.148453] Lustre: Skipped 2 previous similar messages [ 2181.166687] Lustre: Skipped 1 previous similar message [ 2186.222653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2186.230369] Lustre: Skipped 1 previous similar message [ 2186.583883] Lustre: server umount lustre-OST0000 complete [ 2190.818590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2190.829972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.848731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2197.154724] Lustre: server umount lustre-MDT0000 complete [ 2201.568847] LustreError: 74431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.589737] LustreError: 74431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2201.969094] LustreError: 74413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491015 with bad export cookie 2124480278577322890 [ 2201.979748] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.987247] LustreError: 74413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2202.550606] Lustre: server umount lustre-MDT0001 complete [ 2215.197135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2218.912325] Key type lgssc unregistered [ 2219.310345] LNet: 78204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.330689] LNetError: 78204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.391870] LNet: Removed LNI 192.168.204.126@tcp [ 2221.313971] Key type .llcrypt unregistered [ 2221.318547] Key type ._llcrypt unregistered [ 2248.677492] Key type ._llcrypt registered [ 2248.680973] Key type .llcrypt registered [ 2248.796784] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2249.619632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.668809] alg: No test for adler32 (adler32-zlib) [ 2250.854394] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2251.125919] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2252.823293] Key type lgssc registered [ 2254.017869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2261.790632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2261.814483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.315359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.113226] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2273.762114] LustreError: 79614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.780446] LustreError: 79614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2276.995445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.526982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2281.954253] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2289.328924] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2295.187232] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2302.027765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2302.413609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.181879] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2315.248989] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2321.689603] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2322.650332] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2324.562849] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2324.569336] Lustre: Skipped 1 previous similar message [ 2326.963891] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2326.974191] Lustre: Skipped 2 previous similar messages [ 2331.097416] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2331.106273] Lustre: Skipped 4 previous similar messages [ 2339.555932] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2339.562966] Lustre: Skipped 9 previous similar messages [ 2355.734152] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2355.743698] Lustre: Skipped 19 previous similar messages [ 2388.497992] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2388.506976] Lustre: Skipped 38 previous similar messages [ 2414.828123] Lustre: 87709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2417.780347] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:53:48 (1789491228) [ 2420.401454] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2422.191099] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:53:53 (1789491233) [ 2426.337766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2426.343158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.358551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.360630] Lustre: Skipped 1 previous similar message [ 2426.369132] Lustre: Skipped 1 previous similar message [ 2431.458854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.466682] Lustre: Skipped 1 previous similar message [ 2432.368848] Lustre: server umount lustre-OST0000 complete [ 2436.065995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2436.077624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.095900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.697292] LustreError: 80347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.740441] Lustre: server umount lustre-MDT0000 complete [ 2445.275101] LustreError: 79593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491259 with bad export cookie 8984150193312958805 [ 2445.284470] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2445.290057] LustreError: 79593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2445.604542] Lustre: server umount lustre-MDT0001 complete [ 2455.763694] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2458.600689] Key type lgssc unregistered [ 2458.993486] LNet: 88956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.003189] LNetError: 88956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.035497] LNet: Removed LNI 192.168.204.126@tcp [ 2459.838990] Key type .llcrypt unregistered [ 2459.841775] Key type ._llcrypt unregistered [ 2464.384021] Key type ._llcrypt registered [ 2464.385560] Key type .llcrypt registered [ 2464.524912] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2468.804176] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules [ 2469.644771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2469.662595] alg: No test for adler32 (adler32-zlib) [ 2470.697153] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2471.028920] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2472.735375] Key type lgssc registered [ 2474.039385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.105971] LustreError: 90153:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2492.539942] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2497.441702] LustreError: 90153:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2501.955752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2501.987104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.537698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.451712] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2513.891577] LustreError: 91178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.909663] LustreError: 91178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2515.974715] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:55:27 (1789491327) [ 2519.011085] LustreError: 91179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.929327] LustreError: 91210:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2520.940561] LustreError: 91210:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2520.953353] LustreError: 91210:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2521.319764] Lustre: server umount lustre-MDT0000 complete [ 2538.039281] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2542.822667] Key type lgssc unregistered [ 2543.269731] LNet: 92704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.284897] LNetError: 92704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.372566] LNet: Removed LNI 192.168.204.126@tcp [ 2545.064657] Key type .llcrypt unregistered [ 2545.068470] Key type ._llcrypt unregistered [ 2564.417074] Key type ._llcrypt registered [ 2564.420767] Key type .llcrypt registered [ 2564.527898] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2565.448505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2565.542638] alg: No test for adler32 (adler32-zlib) [ 2566.655804] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2566.912892] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2568.577939] Key type lgssc registered [ 2569.628711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.486401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.506559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.079371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.467890] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2588.653099] LustreError: 93921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.678222] LustreError: 93921:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2591.402731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.127066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.450148] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2603.635927] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2609.501464] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2616.631603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.783607] LustreError: 95749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.244764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.959456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2630.282865] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2677.308551] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:58:08 (1789491488) [ 2679.176682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2681.278354] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:58:12 (1789491492) [ 2688.994880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2689.015181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.055114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.020643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.022841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.031263] Lustre: Skipped 1 previous similar message [ 2690.036542] Lustre: Skipped 1 previous similar message [ 2692.173392] Lustre: server umount lustre-MDT0000 complete [ 2695.156697] LustreError: 95770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.219610] LustreError: 95770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2697.091276] LustreError: 93900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491510 with bad export cookie 14704409658432818785 [ 2697.110596] LustreError: 93900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2697.111364] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.654542] Lustre: server umount lustre-MDT0001 complete [ 2712.969701] Lustre: server umount lustre-OST0000 complete [ 2729.738450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.198395] LustreError: 98193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.385258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.220862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2735.589383] LustreError: 98194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.818164] LustreError: 98194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.856212] LustreError: 98194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2747.353414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.409249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2753.759968] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2764.114744] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2773.128494] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2787.137307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2787.391888] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2787.401575] LustreError: 98181:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2787.447683] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2787.460828] LustreError: 100278:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2787.467835] LustreError: 100278:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2787.474258] LustreError: 100278:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2787.607785] Lustre: server umount lustre-OST0000 complete [ 2787.613749] LustreError: 100278:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2795.722216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.935715] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2795.947725] LustreError: 98181:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2795.994311] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2796.006846] LustreError: 100744:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2796.019863] LustreError: 100744:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2796.027717] LustreError: 100744:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2796.108806] Lustre: server umount lustre-OST0000 complete [ 2796.120474] LustreError: 100744:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2805.042689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.234265] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2805.238427] Lustre: Found index 0 for lustre-OST0000, updating log [ 2805.247926] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2805.333344] Lustre: lustre-OST0000: new disk, initializing [ 2805.337917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2805.345458] Lustre: 101280:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2805.690864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.780270] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2813.684839] Lustre: Failing over lustre-OST0000 [ 2813.830566] Lustre: server umount lustre-OST0000 complete [ 2818.204308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.212447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.222850] Lustre: Skipped 1 previous similar message [ 2818.241265] Lustre: Skipped 1 previous similar message [ 2823.650344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2823.659770] Lustre: Skipped 1 previous similar message [ 2824.752831] Lustre: server umount lustre-MDT0000 complete [ 2828.772844] LustreError: 98194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.824785] LustreError: 98194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2829.186664] LustreError: 98174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491642 with bad export cookie 14704409658432821529 [ 2829.190778] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2829.217294] LustreError: 98174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2830.077555] Lustre: server umount lustre-MDT0001 complete [ 2840.475308] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:00:51 (1789491651) [ 2857.946609] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2867.856497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.269296] LustreError: 103639:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.375178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.896184] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2881.609642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.908915] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2891.355554] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2900.075624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.716346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2900.724149] Lustre: Skipped 1 previous similar message [ 2907.782463] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2916.577355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2916.672524] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2916.770370] Lustre: lustre-OST0001: new disk, initializing [ 2916.773322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2916.785760] Lustre: 105956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2923.532642] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2927.215656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2927.226445] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2927.338233] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2931.408869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2944.297540] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2949.089716] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2949.101174] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.114845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2953.186035] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.211091] Lustre: Skipped 1 previous similar message [ 2955.157966] Lustre: server umount lustre-MDT0001 complete [ 2956.876736] LustreError: 105969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.893164] LustreError: 105969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2964.307550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.420947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2964.427925] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2964.475214] Lustre: 103628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2965.122866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2965.131101] Lustre: Skipped 1 previous similar message [ 2966.389389] LustreError: 107540:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2966.398156] LustreError: 107540:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2966.410044] LustreError: 103631:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2966.528112] LustreError: 107548:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.126@tcp: cfg command failed: rc = -17 [ 2966.540490] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2966.540490] [ 2969.706700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2970.605719] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2979.133977] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:03:10 (1789491790) [ 2998.768435] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3015.951332] Lustre: 109573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3038.886503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3042.732766] Lustre: 110707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3128.801993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3128.812727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.825655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.829499] Lustre: Skipped 4 previous similar messages [ 3134.665431] Lustre: server umount lustre-MDT0000 complete [ 3138.559593] LustreError: 103620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491952 with bad export cookie 14704409658432822523 [ 3138.563618] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3138.581165] LustreError: 103620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3139.127867] Lustre: server umount lustre-MDT0001 complete [ 3152.598460] Lustre: server umount lustre-OST0000 complete [ 3155.423326] Lustre: 93231:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789491953/real 1789491953] req@ffff9929ccf52d80 x1876417713636608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789491969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3155.453330] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.466749] Lustre: Skipped 3 previous similar messages [ 3156.308326] Lustre: server umount lustre-OST0001 complete [ 3162.178987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3173.391509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3182.822583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3197.747574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3221.730671] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3233.900961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.924277] Lustre: lustre-MDT0000: reset Object Index mappings [ 3234.431594] LustreError: 114658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.451116] LustreError: 114658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3234.544780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.100620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3248.579851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.609216] Lustre: lustre-MDT0001: reset Object Index mappings [ 3253.929834] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3257.129952] Lustre: 115807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3266.065770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.139285] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3274.030026] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3280.383803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3283.292308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.608325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3289.623221] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3291.783148] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3300.005878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3303.596204] Lustre: 117686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3357.060287] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:09:27 (1789492167) [ 3381.744535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.755518] Lustre: Skipped 1 previous similar message [ 3381.768037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3381.780540] Lustre: Skipped 10 previous similar messages [ 3388.169670] Lustre: server umount lustre-MDT0000 complete [ 3391.766143] LustreError: 114640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492205 with bad export cookie 14704409658432869437 [ 3391.773058] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3391.776627] LustreError: 114640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3406.303837] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3406.549887] Lustre: server umount lustre-MDT0001 complete [ 3410.388510] LustreError: 118451:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0xeb:0x0].0x0 (ffff992ad0528b00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3416.573917] Lustre: server umount lustre-OST0000 complete [ 3426.791086] Lustre: server umount lustre-OST0001 complete [ 3434.969862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3446.438176] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3494.917221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.213807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3495.254750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3495.370917] Lustre: lustre-MDT0000: new disk, initializing [ 3495.506717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.514572] Lustre: Skipped 3 previous similar messages [ 3495.541271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3500.876556] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3508.178698] Lustre: server umount lustre-MDT0000 complete [ 3519.543560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.829806] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3519.868375] Lustre: 121967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3519.890347] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3525.627531] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3531.236840] Lustre: server umount lustre-MDT0000 complete [ 3539.756472] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:12:31 (1789492351) [ 3541.450696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3543.502790] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:12:34 (1789492354) [ 3563.548837] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3575.116840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.814683] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3588.957505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.084070] Lustre: 124017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3589.153692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3589.156949] Lustre: Skipped 1 previous similar message [ 3589.246887] Lustre: lustre-MDT0001: new disk, initializing [ 3589.345862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3589.355829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3594.063317] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3599.403928] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3606.953197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3607.297486] Lustre: lustre-OST0000: new disk, initializing [ 3607.301497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3607.307911] Lustre: 125594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3608.849926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3608.868744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3609.076066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3615.761271] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3627.336894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.515244] Lustre: lustre-OST0001: new disk, initializing [ 3627.520691] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3627.537087] Lustre: 126467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3629.279190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3629.296821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3629.406818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3635.475320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3647.206511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3652.049861] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4060.512363] Lustre: 132885:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4060.520083] Lustre: 132885:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4060.526967] Lustre: 132885:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4060.536343] Lustre: 132885:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4060.544832] Lustre: 132885:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4060.555843] Lustre: 132885:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4347.778826] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:25:58 (1789493158) [ 4362.720834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4362.732799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.747233] Lustre: Skipped 4 previous similar messages [ 4362.756857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4362.764384] Lustre: Skipped 10 previous similar messages [ 4366.821831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4366.834752] Lustre: Skipped 1 previous similar message [ 4368.931763] Lustre: server umount lustre-MDT0000 complete [ 4371.940196] LustreError: 124028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.958175] LustreError: 124028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 4372.873886] LustreError: 124009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789493186 with bad export cookie 14704409658432937582 [ 4372.885957] LustreError: 124009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4372.889770] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4377.057402] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.070915] Lustre: Skipped 3 previous similar messages [ 4377.079708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4377.086641] Lustre: Skipped 1 previous similar message [ 4387.295336] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4387.815125] Lustre: server umount lustre-MDT0001 complete [ 4397.757390] Lustre: server umount lustre-OST0000 complete [ 4407.736874] Lustre: server umount lustre-OST0001 complete [ 4415.205303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4424.768906] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4461.483499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.900733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4461.960235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4462.076657] Lustre: lustre-MDT0000: new disk, initializing [ 4462.254689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.266612] Lustre: Skipped 5 previous similar messages [ 4462.283885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4466.948334] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4478.727872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.949297] Lustre: 135947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4479.032691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4479.046545] Lustre: Skipped 1 previous similar message [ 4479.163237] Lustre: lustre-MDT0001: new disk, initializing [ 4479.340222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4479.347561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4484.461618] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4493.035771] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.852622] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4504.019525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.339162] Lustre: lustre-OST0000: new disk, initializing [ 4504.342212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4504.351785] Lustre: 137920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4506.188479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4506.201904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4506.294954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4509.727890] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4518.303668] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4527.807679] Lustre: lustre-MDD0000: changelog on [ 4530.894307] Lustre: lustre-MDD0001: changelog on [ 6378.557802] Lustre: lustre-MDD0001: changelog off [ 6380.505628] Lustre: 135956:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6381.438076] Lustre: lustre-MDD0000: changelog off [ 6390.755301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6390.762073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.771659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6390.781118] Lustre: Skipped 2 previous similar messages [ 6390.789366] Lustre: Skipped 4 previous similar messages [ 6392.948079] Lustre: server umount lustre-OST0000 complete [ 6399.463663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6399.471261] LustreError: Skipped 1 previous similar message [ 6399.479774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6399.499309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6399.506207] Lustre: Skipped 1 previous similar message [ 6406.114931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6406.122376] Lustre: Skipped 3 previous similar messages [ 6412.255140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6412.741660] Lustre: server umount lustre-MDT0000 complete [ 6416.358049] LustreError: 135961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.390586] LustreError: 135961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6416.477513] LustreError: 137919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789495230 with bad export cookie 14704409658433762238 [ 6416.483639] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.489031] LustreError: 137919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6416.841403] Lustre: server umount lustre-MDT0001 complete [ 6428.706442] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6431.443241] Key type lgssc unregistered [ 6431.798651] LNet: 147045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6431.807380] LNetError: 147045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6431.846544] LNet: Removed LNI 192.168.204.126@tcp [ 6432.868133] Key type .llcrypt unregistered [ 6432.874677] Key type ._llcrypt unregistered [ 6453.043991] Key type ._llcrypt registered [ 6453.046761] Key type .llcrypt registered [ 6453.186199] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6468.433878] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6469.501068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6469.585088] alg: No test for adler32 (adler32-zlib) [ 6470.720667] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 6471.123079] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6472.871275] Key type lgssc registered [ 6473.914628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6506.406796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6506.449375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.678260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6507.715302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6507.811756] Lustre: lustre-MDT0000: new disk, initializing [ 6507.922080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.951533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6512.629878] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6523.166990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.289994] Lustre: 150254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6523.341930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6523.345799] Lustre: Skipped 1 previous similar message [ 6523.414849] Lustre: lustre-MDT0001: new disk, initializing [ 6523.496892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.530828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6523.545550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6528.016339] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6536.250341] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6542.121899] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6548.491077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.717325] Lustre: lustre-OST0000: new disk, initializing [ 6548.721513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6548.730160] Lustre: 152220:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6548.819567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.095304] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6556.237930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6556.249320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6556.411951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6564.765671] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6571.515559] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_module obdecho/obdecho [ 6573.224885] LustreError: 153222:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6573.236282] LustreError: 153222:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6574.897753] LustreError: 153318:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6587.362294] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.367797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6587.390867] Lustre: Skipped 1 previous similar message [ 6587.406200] Lustre: Skipped 2 previous similar messages [ 6592.483854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6592.492575] Lustre: Skipped 2 previous similar messages [ 6593.385311] Lustre: server umount lustre-MDT0000 complete [ 6597.539344] LustreError: 150245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789495411 with bad export cookie 9371427877211352453 [ 6597.549118] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6597.557663] LustreError: 150245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6597.605465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.620230] Lustre: Skipped 1 previous similar message [ 6597.635639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6597.843804] Lustre: server umount lustre-MDT0001 complete [ 6603.868298] Lustre: server umount lustre-OST0000 complete [ 6627.445314] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6640.431527] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6679.693948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.941641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6679.971220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6680.036252] Lustre: lustre-MDT0000: new disk, initializing [ 6680.106365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6680.133982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6684.844741] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6696.147087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6696.318226] Lustre: 157038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6696.390759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6696.394632] Lustre: Skipped 1 previous similar message [ 6696.508570] Lustre: lustre-MDT0001: new disk, initializing [ 6696.682713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.755052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6696.764706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6701.682766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6710.943745] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6716.482414] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6724.028966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6724.665148] Lustre: lustre-OST0000: new disk, initializing [ 6724.686736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6724.706397] Lustre: 159006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6724.938595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.945349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6733.963943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6734.201314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6734.734931] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6747.782180] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6753.964512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6754.313544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6760.332925] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6760.662660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6765.024296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6765.029905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6765.046797] LustreError: Skipped 1 previous similar message [ 6765.054209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6765.080120] Lustre: Skipped 1 previous similar message [ 6765.115618] Lustre: Skipped 1 previous similar message [ 6769.834752] Lustre: server umount lustre-OST0000 complete [ 6775.269238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.301490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6775.319470] Lustre: Skipped 1 previous similar message [ 6783.829603] Lustre: server umount lustre-MDT0000 complete [ 6784.483984] LustreError: 158408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.506483] LustreError: 158408:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6787.799552] LustreError: 157031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789495601 with bad export cookie 9371427877211353412 [ 6787.809894] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6787.817053] LustreError: 157031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6788.180386] Lustre: server umount lustre-MDT0001 complete [ 6792.596965] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:06:43 (1789495603) [ 6794.647632] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6796.910304] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:06:47 (1789495607) [ 6805.445090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.966983] LustreError: 161421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.064358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6811.086964] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6811.108765] LustreError: 161422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.138903] LustreError: 161422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6816.230311] LustreError: 161421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.347194] LustreError: 161422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.343756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.424664] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6838.441959] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6845.066906] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6852.437979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6852.926542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6852.940756] Lustre: Skipped 1 previous similar message [ 6860.560534] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6867.862502] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6885.158171] Lustre: server umount lustre-MDT0000 complete [ 6885.346034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6885.364192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.402095] Lustre: Skipped 1 previous similar message [ 6885.415642] LustreError: 162150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6889.323099] LustreError: 161402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789495703 with bad export cookie 9371427877211354294 [ 6889.335076] LustreError: 161402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6889.339682] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.983980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6890.991980] Lustre: Skipped 2 previous similar messages [ 6895.768947] Lustre: server umount lustre-MDT0001 complete [ 6901.896535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6918.379432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.875544] LustreError: 165284:0:(llog.c:1655:llog_backup()) MGC192.168.204.126@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6918.890062] Lustre: 165284:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.126@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6919.652033] LustreError: 165289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6919.682254] LustreError: 165289:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6955.084815] LustreError: 165289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6955.094881] LustreError: 165289:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 23 previous similar messages [ 6979.552721] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6979.566746] Lustre: 165284:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7019.596878] LustreError: 165290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7019.622963] LustreError: 165290:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 7020.542817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7025.661836] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7026.224858] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7035.448065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.879902] LustreError: 148474:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003f0fb79e x1876421807050368/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7041.294230] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7041.524908] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7049.503552] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7051.126453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7056.992571] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7058.497697] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7077.346988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7077.350867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.358776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.358786] Lustre: Skipped 2 previous similar messages [ 7077.398783] Lustre: Skipped 5 previous similar messages [ 7082.459740] Lustre: server umount lustre-MDT0000 complete [ 7087.381116] LustreError: 161401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789495901 with bad export cookie 9371427877211360748 [ 7087.383911] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.395210] LustreError: 161401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7087.966980] Lustre: server umount lustre-MDT0001 complete [ 7092.901393] Lustre: server umount lustre-OST0000 complete [ 7115.672516] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7127.476108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7167.599211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.925108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7167.961621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7168.037466] Lustre: lustre-MDT0000: new disk, initializing [ 7168.132264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7168.137224] Lustre: Skipped 1 previous similar message [ 7168.162918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7173.174355] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7184.735957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.878220] Lustre: 170392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7185.180841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7190.759590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7202.413313] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7208.626164] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7216.685762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.348509] Lustre: lustre-OST0000: new disk, initializing [ 7217.357104] Lustre: Skipped 1 previous similar message [ 7217.373520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7217.383506] Lustre: Skipped 2 previous similar messages [ 7217.395354] Lustre: 172363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7225.822950] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7226.505532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7226.519891] Lustre: Skipped 1 previous similar message [ 7226.533665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7226.777179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7238.684520] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7244.825715] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7245.028787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7250.006564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7250.283434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7257.587212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.604531] Lustre: Skipped 1 previous similar message [ 7257.606666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7257.607929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7257.621238] Lustre: Skipped 4 previous similar messages [ 7259.832587] Lustre: server umount lustre-OST0000 complete [ 7267.314713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7267.325496] LustreError: Skipped 1 previous similar message [ 7272.146182] Lustre: server umount lustre-MDT0000 complete [ 7272.937292] LustreError: 170403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.963858] LustreError: 170403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 7276.669641] LustreError: 170384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496090 with bad export cookie 9371427877211377583 [ 7276.690529] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7276.959720] Lustre: server umount lustre-MDT0001 complete [ 7288.445605] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:14:59 (1789496099) [ 7296.088631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.999652] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7313.291454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.789201] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7326.996859] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7333.542618] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7341.089689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.041535] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7356.390352] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7372.342501] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:16:23 (1789496183) [ 7377.378626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7377.404423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.422560] Lustre: Skipped 3 previous similar messages [ 7377.427485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7377.445987] Lustre: Skipped 2 previous similar messages [ 7382.735288] Lustre: server umount lustre-OST0000 complete [ 7393.092561] Lustre: server umount lustre-MDT0000 complete [ 7397.358648] LustreError: 174758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496211 with bad export cookie 9371427877211378458 [ 7397.373889] LustreError: 174758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7397.381259] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7397.948680] Lustre: server umount lustre-MDT0001 complete [ 7411.084083] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7414.718541] Key type lgssc unregistered [ 7415.102250] LNet: 178508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7415.130183] LNetError: 178508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7416.167288] LNet: Removed LNI 192.168.204.126@tcp [ 7417.013145] Key type .llcrypt unregistered [ 7417.015108] Key type ._llcrypt unregistered [ 7440.508848] Key type ._llcrypt registered [ 7440.512105] Key type .llcrypt registered [ 7440.671366] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7441.936474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.024665] alg: No test for adler32 (adler32-zlib) [ 7443.403146] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7443.736953] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7445.567396] Key type lgssc registered [ 7446.946119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7455.388431] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7455.419173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7471.013326] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7476.191436] LustreError: 179747:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 7481.319972] LustreError: 179747:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7481.336341] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7481.346199] LustreError: MGC192.168.204.126@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7481.359288] LustreError: 179747:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7481.373939] LustreError: 179747:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7481.389993] LustreError: 179747:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7481.523546] Lustre: server umount lustre-OST0000 complete [ 7481.528493] LustreError: 179747:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7490.423724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.090429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.617687] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7501.286043] LustreError: 180246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.302722] LustreError: 180246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7505.041675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.523102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7509.719806] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7518.663775] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7525.795546] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7535.424597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7535.960826] LustreError: 182063:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7535.972545] Lustre: 182063:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.126@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7536.137212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7544.710585] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7553.132205] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7555.560032] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.562069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7555.577574] Lustre: Skipped 2 previous similar messages [ 7555.601714] Lustre: Skipped 2 previous similar messages [ 7560.681653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7560.691600] Lustre: Skipped 2 previous similar messages [ 7561.773637] Lustre: server umount lustre-MDT0000 complete [ 7565.800064] LustreError: 182088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.831550] LustreError: 182088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7566.172686] LustreError: 180227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496379 with bad export cookie 2230397860283199870 [ 7566.189077] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7566.730869] Lustre: server umount lustre-MDT0001 complete [ 7580.571321] Lustre: server umount lustre-OST0000 complete [ 7592.360856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7607.462527] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7652.153316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.626432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7652.706593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7652.774587] Lustre: lustre-MDT0000: new disk, initializing [ 7652.855426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7652.873139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7658.420314] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7671.708804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.801204] Lustre: 185611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7671.845756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7671.848293] Lustre: Skipped 1 previous similar message [ 7671.910525] Lustre: lustre-MDT0001: new disk, initializing [ 7671.995656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7672.026981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7672.044492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7677.049598] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7688.199914] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7694.244718] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7700.424662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.714488] Lustre: lustre-OST0000: new disk, initializing [ 7700.717991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7700.726483] Lustre: 187576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7700.874251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.758390] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7708.710071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7708.723432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7708.838432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7716.372635] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7721.066629] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7721.288882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7726.908092] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7727.141552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7729.123369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7729.132615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7729.142317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7729.633124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.193604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.200471] Lustre: Skipped 2 previous similar messages [ 7735.341116] Lustre: server umount lustre-OST0000 complete [ 7742.432305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.448306] Lustre: Skipped 1 previous similar message [ 7742.451566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.456266] Lustre: Skipped 1 previous similar message [ 7745.782546] Lustre: server umount lustre-MDT0000 complete [ 7747.556419] LustreError: 185622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.593733] LustreError: 185622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7749.557581] LustreError: 185600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496563 with bad export cookie 2230397860283200591 [ 7749.559856] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.566574] LustreError: 185600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7750.141840] Lustre: server umount lustre-MDT0001 complete [ 7759.783595] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:22:50 (1789496570) [ 7778.094232] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7781.225219] Key type lgssc unregistered [ 7781.498616] LNet: 190523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7781.519232] LNetError: 190523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7781.545819] LNet: Removed LNI 192.168.204.126@tcp [ 7782.396568] Key type .llcrypt unregistered [ 7782.402316] Key type ._llcrypt unregistered [ 7802.085802] Key type ._llcrypt registered [ 7802.088734] Key type .llcrypt registered [ 7802.285856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7803.735534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7803.779416] alg: No test for adler32 (adler32-zlib) [ 7804.903280] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7805.256646] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7807.008403] Key type lgssc registered [ 7808.728929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7817.378384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7817.409784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.913817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.890196] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7828.452231] LustreError: 191740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.471345] LustreError: 191740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7833.568093] LustreError: 191741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.782358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.143280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.158447] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7848.444103] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7855.461954] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7866.349790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7866.943716] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7866.949292] LustreError: 193604:0:(llog.c:1696:llog_backup()) MGC192.168.204.126@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7866.961419] Lustre: 193604:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.126@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7867.227217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.573450] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7882.687300] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7885.281534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7885.292591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.302604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.870024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.872783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.912293] Lustre: Skipped 2 previous similar messages [ 7891.589123] Lustre: server umount lustre-MDT0000 complete [ 7893.985886] LustreError: 191741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7894.008782] LustreError: 191741:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7895.775697] LustreError: 191722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496709 with bad export cookie 5877318663357263916 [ 7895.788317] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7895.803244] LustreError: 191722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7896.398444] Lustre: server umount lustre-MDT0001 complete [ 7911.912484] Lustre: server umount lustre-OST0000 complete [ 7925.566774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7941.424283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7978.315616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.545297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7978.583763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7978.681553] Lustre: lustre-MDT0000: new disk, initializing [ 7978.741328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.760428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7983.466940] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7993.747554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.877543] Lustre: 197150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7993.932857] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7993.935371] Lustre: Skipped 1 previous similar message [ 7993.944541] LustreError: 197933:0:(llog.c:1696:llog_backup()) MGC192.168.204.126@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7993.961032] LustreError: 197933:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7993.966594] Lustre: 197933:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.126@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7993.991343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7993.995979] Lustre: Skipped 1 previous similar message [ 7994.154104] Lustre: lustre-MDT0001: new disk, initializing [ 7994.266906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7994.299082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7994.321564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7998.116186] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8006.598936] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8012.503390] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8020.554735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8020.968669] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8020.970860] Lustre: Skipped 1 previous similar message [ 8020.978758] LustreError: 199095:0:(llog.c:1696:llog_backup()) MGC192.168.204.126@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8020.990791] LustreError: 199095:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8020.996516] Lustre: 199095:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.126@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8021.010655] Lustre: 199095:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8021.052027] Lustre: lustre-OST0000: new disk, initializing [ 8021.062504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8021.081883] Lustre: 199119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8021.241337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8028.400573] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8029.316920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8029.336247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8029.566851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8041.697810] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8046.545326] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8046.753470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8051.922723] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8052.197690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8055.265178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.270034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8055.287180] Lustre: Skipped 1 previous similar message [ 8055.290823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8055.310557] Lustre: Skipped 1 previous similar message [ 8060.390107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8060.402994] Lustre: Skipped 1 previous similar message [ 8060.648736] Lustre: server umount lustre-OST0000 complete [ 8065.507063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.520151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8065.528033] Lustre: Skipped 2 previous similar messages [ 8065.545925] Lustre: Skipped 1 previous similar message [ 8075.746683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8075.751589] Lustre: Skipped 3 previous similar messages [ 8079.328648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8079.778542] Lustre: server umount lustre-MDT0000 complete [ 8080.867744] LustreError: 198466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.888899] LustreError: 198466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8083.893252] LustreError: 197143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496897 with bad export cookie 5877318663357264630 [ 8083.901151] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8083.913417] LustreError: 197143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8084.628277] Lustre: server umount lustre-MDT0001 complete [ 8095.982205] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:28:27 (1789496907) [ 8113.861871] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8123.145444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.707409] LustreError: 202250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.745389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.070987] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8134.118501] LustreError: 202251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8134.151083] LustreError: 202251:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8137.432503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.846699] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8144.537310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8152.079921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8159.311874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8168.861815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.087313] Lustre: lustre-OST0001: new disk, initializing [ 8169.090601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8169.102664] Lustre: 204559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8169.234942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8169.245113] Lustre: Skipped 2 previous similar messages [ 8175.703526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8175.711098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8175.810191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8176.039770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8186.855650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8193.820648] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8197.004683] Lustre: DEBUG MARKER: ADD OST3 [ 8203.543113] loop: module loaded [ 8203.639444] loop0: detected capacity change from 0 to 204800000 [ 8203.960352] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8203.972864] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8203.984318] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8211.207136] Lustre: DEBUG MARKER: STOP OST3 [ 8211.465258] loop0: detected capacity change from 0 to 204800000 [ 8218.185345] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8218.332799] Lustre: lustre-OST0003: new disk, initializing [ 8218.336418] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8218.342433] Lustre: 206602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8220.471883] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8220.625352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8220.629704] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8220.680928] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8222.027392] Lustre: Failing over lustre-OST0003 [ 8222.451485] Lustre: server umount lustre-OST0003 complete [ 8222.496073] LustreError: 206756:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8225.434441] LustreError: 203735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.782300] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8225.783756] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.822371] Lustre: Skipped 1 previous similar message [ 8226.880459] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8234.981991] Lustre: DEBUG MARKER: START OST3 again [ 8241.554112] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.775095] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8241.800795] Lustre: Skipped 1 previous similar message [ 8241.812256] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8243.431017] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8243.664734] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8243.666727] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8248.897760] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8269.371054] Lustre: Failing over lustre-OST0003 [ 8269.449535] Lustre: server umount lustre-OST0003 complete [ 8272.364936] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.389835] LustreError: 204884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8272.401844] LustreError: 204884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8287.728276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8287.736805] Lustre: Skipped 2 previous similar messages [ 8293.218926] Lustre: server umount lustre-MDT0000 complete [ 8297.190229] LustreError: 202231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789497110 with bad export cookie 5877318663357265512 [ 8297.196150] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8297.211258] LustreError: 202231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8297.970422] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.988463] Lustre: Skipped 6 previous similar messages [ 8303.873310] Lustre: server umount lustre-MDT0001 complete [ 8308.176420] Lustre: server umount lustre-OST0000 complete [ 8312.599396] Lustre: server umount lustre-OST0001 complete [ 8319.909738] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 8328.282486] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8366.682695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.914441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8366.938404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8367.001578] Lustre: lustre-MDT0000: new disk, initializing [ 8367.074295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8371.642447] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8382.338082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.468841] Lustre: 212014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8382.687197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.697666] Lustre: Skipped 1 previous similar message [ 8382.737430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8388.163932] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8399.505904] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8406.703484] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8414.838959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8415.278824] Lustre: lustre-OST0000: new disk, initializing [ 8415.286223] Lustre: Skipped 1 previous similar message [ 8415.292170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8415.296150] Lustre: Skipped 2 previous similar messages [ 8415.300647] Lustre: 213985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8416.655144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8416.663726] Lustre: Skipped 1 previous similar message [ 8416.670119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8416.808870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8422.377737] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8432.055852] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8437.501980] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8437.794645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8443.458473] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8443.875051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8447.459578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.462295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8447.492734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8447.492742] Lustre: Skipped 9 previous similar messages [ 8452.582070] LustreError: 213970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.590877] LustreError: 213970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8452.683790] Lustre: server umount lustre-OST0000 complete [ 8462.918596] Lustre: server umount lustre-MDT0000 complete [ 8466.150016] LustreError: 212008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789497279 with bad export cookie 5877318663357267808 [ 8466.158726] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8466.173623] LustreError: 212008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8466.443359] Lustre: server umount lustre-MDT0001 complete [ 8470.494912] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:34:41 (1789497281) [ 8487.375409] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 8498.669579] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8536.815205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.046527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8537.068180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8537.135767] Lustre: lustre-MDT0000: new disk, initializing [ 8537.239539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8543.127311] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8554.069658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.214400] Lustre: 218594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8554.499637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8559.327947] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8568.653052] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8574.848728] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8581.673850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.998755] Lustre: 220562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8583.342988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8588.078430] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8598.208396] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8603.333398] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8603.604986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8608.770579] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8609.022412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8613.866482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8613.871941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8613.879614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8613.879626] Lustre: Skipped 5 previous similar messages [ 8613.926675] Lustre: Skipped 4 previous similar messages [ 8625.119161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8625.377459] Lustre: server umount lustre-OST0000 complete [ 8631.268605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8636.251896] Lustre: server umount lustre-MDT0000 complete [ 8639.459594] LustreError: 218606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8639.484268] LustreError: 218606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8640.699218] LustreError: 218587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789497454 with bad export cookie 5877318663357268690 [ 8640.701333] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8640.708557] LustreError: 218587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8641.168962] Lustre: server umount lustre-MDT0001 complete [ 8650.311637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8650.966827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.976842] Lustre: Skipped 4 previous similar messages [ 8656.087025] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8666.033619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.314448] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8680.991530] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8687.277677] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8693.745885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8699.855278] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8707.615730] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8720.870506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8724.261405] Lustre: server umount lustre-OST0000 complete [ 8742.367630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8742.583346] Lustre: server umount lustre-MDT0000 complete [ 8746.391891] LustreError: 222762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789497560 with bad export cookie 5877318663357269572 [ 8746.404514] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8746.407631] LustreError: 222762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8746.745972] Lustre: server umount lustre-MDT0001 complete [ 8848.812939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8851.766405] Key type lgssc unregistered [ 8852.093393] LNet: 226448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8852.108339] LNetError: 226448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8852.134389] LNet: Removed LNI 192.168.204.126@tcp [ 8853.121169] Key type .llcrypt unregistered [ 8853.127606] Key type ._llcrypt unregistered [ 8879.849210] Key type ._llcrypt registered [ 8879.852772] Key type .llcrypt registered [ 8879.999564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 8896.628029] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8899.010254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8899.087942] alg: No test for adler32 (adler32-zlib) [ 8900.181542] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 8900.489901] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8902.191502] Key type lgssc registered [ 8903.496699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8944.229860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8944.282172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8945.542713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8945.591804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8945.667584] Lustre: lustre-MDT0000: new disk, initializing [ 8945.771678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.793929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8950.495928] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8960.611992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.724898] Lustre: 229797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8960.788881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8960.793366] Lustre: Skipped 1 previous similar message [ 8960.856659] Lustre: lustre-MDT0001: new disk, initializing [ 8960.936452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8960.969594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8960.991535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8965.429908] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8974.063275] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8979.569528] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8986.066529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8986.288452] Lustre: lustre-OST0000: new disk, initializing [ 8986.291907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8986.296884] Lustre: 231764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8986.373973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.847637] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8995.418577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8995.428289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8995.555951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9002.670877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9007.848498] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9008.152771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9012.652074] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9012.862727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9016.289765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9016.299878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.317675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9016.325605] Lustre: Skipped 1 previous similar message [ 9021.358687] Lustre: server umount lustre-OST0000 complete [ 9026.529445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.546611] Lustre: Skipped 1 previous similar message [ 9026.551861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.563965] Lustre: Skipped 1 previous similar message [ 9031.649470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9031.658428] Lustre: Skipped 1 previous similar message [ 9036.768924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9036.774135] Lustre: Skipped 1 previous similar message [ 9039.327304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9041.654826] Lustre: server umount lustre-MDT0000 complete [ 9045.659507] LustreError: 229789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789497859 with bad export cookie 4031799675785930778 [ 9045.666303] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9045.688900] LustreError: 229789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9046.035651] Lustre: server umount lustre-MDT0001 complete [ 9066.277770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9076.778345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9077.253584] LustreError: 234700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.270798] LustreError: 234700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9077.337426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.430769] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9082.349097] LustreError: 234701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9087.459342] LustreError: 234700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.279968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.811573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9095.781125] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9098.864722] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9105.357270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9105.709106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9111.510375] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9120.075326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.220788] Lustre: lustre-OST0001: new disk, initializing [ 9120.223708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9120.228590] Lustre: 237007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9126.524220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9126.536550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9126.638974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9126.791383] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9137.846829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9147.681713] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9162.038168] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:46:12 (1789497972) [ 9165.794495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.807940] Lustre: Skipped 1 previous similar message [ 9165.817027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9165.838440] Lustre: Skipped 3 previous similar messages [ 9170.817818] Lustre: server umount lustre-MDT0000 complete [ 9170.924144] LustreError: 234701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.961392] LustreError: 234701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9175.940063] LustreError: 235833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789497989 with bad export cookie 4031799675785931660 [ 9175.958403] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9175.979564] LustreError: 235833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9176.036284] LustreError: 234695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.036831] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.038061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9176.073641] LustreError: 234695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9176.115552] Lustre: Skipped 4 previous similar messages [ 9176.447333] Lustre: server umount lustre-MDT0001 complete [ 9182.826680] Lustre: server umount lustre-OST0000 complete [ 9187.032447] Lustre: server umount lustre-OST0001 complete [ 9193.709687] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9206.190148] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9247.117773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.387431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9247.433690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9247.538444] Lustre: lustre-MDT0000: new disk, initializing [ 9247.608342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9247.613986] Lustre: Skipped 1 previous similar message [ 9247.627792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9252.083467] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9262.026560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.161354] Lustre: 240952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9262.231758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9262.235502] Lustre: Skipped 1 previous similar message [ 9262.350180] Lustre: lustre-MDT0001: new disk, initializing [ 9262.527870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9262.543486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9267.426970] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9276.882889] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9282.227580] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9288.533358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9288.802487] Lustre: lustre-OST0000: new disk, initializing [ 9288.807674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9288.814289] Lustre: 242921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9288.927485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9288.935844] Lustre: Skipped 1 previous similar message [ 9290.142637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9290.153683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9290.295504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9294.900353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9304.046442] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9308.639971] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9308.809848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9313.068688] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9313.279404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9315.814330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9315.820438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.835752] LustreError: Skipped 2 previous similar messages [ 9315.836452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.836456] Lustre: Skipped 1 previous similar message [ 9315.881812] Lustre: Skipped 1 previous similar message [ 9321.291569] Lustre: server umount lustre-OST0000 complete [ 9326.048829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9326.070554] Lustre: Skipped 1 previous similar message [ 9339.359197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9339.591529] Lustre: server umount lustre-MDT0000 complete [ 9341.409229] LustreError: 240963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.422485] LustreError: 240963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9343.984488] LustreError: 240945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789498157 with bad export cookie 4031799675785933802 [ 9343.986374] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9344.014870] LustreError: 240945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9344.413815] Lustre: server umount lustre-MDT0001 complete [ 9354.116388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.752800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9359.846503] LustreError: 245140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.898894] LustreError: 245140:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9361.494682] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9373.721383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.846269] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9387.148110] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9392.856609] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9400.270071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9407.670531] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9414.608294] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9560.549620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9560.558078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9560.578132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9560.604951] Lustre: Skipped 1 previous similar message [ 9560.620953] Lustre: Skipped 10 previous similar messages [ 9574.367144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9574.559475] Lustre: server umount lustre-OST0000 complete [ 9578.979955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9584.325940] Lustre: server umount lustre-MDT0000 complete [ 9586.149434] LustreError: 246984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9586.163568] LustreError: 246984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9588.669869] LustreError: 245121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789498402 with bad export cookie 4031799675785934684 [ 9588.681242] LustreError: 245121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9588.684765] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9588.936745] Lustre: server umount lustre-MDT0001 complete [ 9684.405815] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9687.692354] Key type lgssc unregistered [ 9688.006773] LNet: 248751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9688.015904] LNetError: 248751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9688.047119] LNet: Removed LNI 192.168.204.126@tcp [ 9689.046475] Key type .llcrypt unregistered [ 9689.048839] Key type ._llcrypt unregistered [ 9714.263552] Key type ._llcrypt registered [ 9714.265228] Key type .llcrypt registered [ 9714.385952] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9731.493564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9733.534817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9733.567313] alg: No test for adler32 (adler32-zlib) [ 9734.744159] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 9734.988107] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9736.639944] Key type lgssc registered [ 9738.003653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9793.848677] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9806.206880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9806.251977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9807.608209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9807.671701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9807.821579] Lustre: lustre-MDT0000: new disk, initializing [ 9807.918768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9807.951314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9813.487299] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9824.561888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.690200] Lustre: 252836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9824.732465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9824.737304] Lustre: Skipped 1 previous similar message [ 9824.832621] Lustre: lustre-MDT0001: new disk, initializing [ 9824.967636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9825.031281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9825.050324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9829.605453] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9834.448309] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9840.831762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9841.109379] Lustre: lustre-OST0000: new disk, initializing [ 9841.112879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9841.119735] Lustre: 254468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9841.230670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9843.626908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9843.640187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9843.822842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9848.157465] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9860.551525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9860.785649] Lustre: lustre-OST0001: new disk, initializing [ 9860.797788] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9860.807404] Lustre: 255343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9860.911266] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9867.105828] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9871.001255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9871.025030] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9871.157845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9878.806021] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9888.095678] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9900.207304] Lustre: server umount lustre-MDT0000 complete [ 9902.068695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.075788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9903.839733] LustreError: 255342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789498717 with bad export cookie 3453999728380239738 [ 9903.845100] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9903.845477] LustreError: 255342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9904.182983] Lustre: server umount lustre-MDT0001 complete [ 9920.044171] Lustre: server umount lustre-OST0000 complete [ 9922.532077] Lustre: 250326:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789498720/real 1789498720] req@ffff9929c9b55500 x1876425229545984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789498736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9922.549050] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9922.563424] Lustre: Skipped 3 previous similar messages [ 9923.551150] Lustre: 250325:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789498721/real 1789498721] req@ffff992aedac6d80 x1876425229546240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789498737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9924.491790] Lustre: server umount lustre-OST0001 complete [ 9937.599460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9947.025035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9956.609824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9968.622548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9992.035408] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10003.677023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10003.705997] Lustre: lustre-MDT0000: reset Object Index mappings [10004.110965] LustreError: 260354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10004.126101] LustreError: 260354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10004.207180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10008.530541] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10009.572895] LustreError: 260355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.672040] LustreError: 260354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10017.360057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.374759] Lustre: lustre-MDT0001: reset Object Index mappings [10017.813741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10022.667688] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10025.815182] Lustre: 261496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10032.783356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10039.593839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10042.423643] LustreError: 261852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.465146] LustreError: 261851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.308855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.601146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10048.674148] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10048.688093] Lustre: Skipped 1 previous similar message [10054.133705] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10055.379626] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10062.904979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10066.758124] Lustre: 263379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10073.696803] Lustre: server umount lustre-MDT0000 complete [10074.084953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10074.089964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10074.100238] LustreError: 261585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.890755] LustreError: 260336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789498891 with bad export cookie 3453999728380246528 [10077.900744] LustreError: 260336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10077.903812] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10078.436636] Lustre: server umount lustre-MDT0001 complete [10092.937738] Lustre: server umount lustre-OST0000 complete [10095.903240] Lustre: 250324:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789498893/real 1789498893] req@ffff992aed8c1880 x1876425229593216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789498909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10095.948437] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10095.965974] Lustre: Skipped 3 previous similar messages [10097.151372] Lustre: server umount lustre-OST0001 complete [10118.591254] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:02:09 (1789498929) [10135.223556] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [10146.337650] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10182.370296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10182.713205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10182.750596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10182.903572] Lustre: lustre-MDT0000: new disk, initializing [10183.002856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10183.031777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10187.963135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10199.824261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.946871] Lustre: 267634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10199.993857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10199.997385] Lustre: Skipped 1 previous similar message [10200.069442] Lustre: lustre-MDT0001: new disk, initializing [10200.284678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10200.295217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10205.555299] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10215.296827] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10221.986424] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10229.138051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.445730] Lustre: lustre-OST0000: new disk, initializing [10229.449328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10229.458325] Lustre: 269603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10231.459719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10231.476565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10231.638430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10236.749557] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10247.252913] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10251.964641] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10252.199973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10257.387559] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10257.698683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10262.503602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10262.521450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10262.542544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10262.569756] Lustre: Skipped 1 previous similar message [10265.661082] Lustre: server umount lustre-OST0000 complete [10272.224385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10272.239032] LustreError: Skipped 1 previous similar message [10272.251032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10272.269620] Lustre: Skipped 1 previous similar message [10272.279423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10276.733973] Lustre: server umount lustre-MDT0000 complete [10277.867075] LustreError: 267640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10277.915508] LustreError: 267640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10282.441157] LustreError: 269602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789499096 with bad export cookie 3453999728380248621 [10282.450713] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10282.462406] LustreError: 269602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10283.087439] Lustre: server umount lustre-MDT0001 complete [10305.154602] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10315.474287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.898451] LustreError: 272538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.933202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10315.941573] Lustre: Skipped 2 previous similar messages [10320.406648] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10329.756692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.532252] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10338.842728] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10346.961767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10356.715922] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10366.126219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10366.315216] Lustre: lustre-OST0001: new disk, initializing [10366.322281] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10366.335913] Lustre: 274847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10367.602944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10367.613435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10367.769346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10372.789654] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10383.589866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10387.155431] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10392.214017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10392.416728] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10402.845306] Lustre: server umount lustre-MDT0000 complete [10403.813676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.824362] LustreError: 272535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10403.829713] Lustre: Skipped 3 previous similar messages [10403.871230] LustreError: 272535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10406.714260] LustreError: 272519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789499220 with bad export cookie 3453999728380249503 [10406.719485] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10406.727078] LustreError: 272519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10407.031856] Lustre: server umount lustre-MDT0001 complete [10420.485099] Lustre: server umount lustre-OST0000 complete [10434.453833] Lustre: server umount lustre-OST0001 complete [10452.628680] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10464.413377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.917030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10464.928737] Lustre: Skipped 3 previous similar messages [10469.294620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10477.918041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.946456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10486.262706] Lustre: 279331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10492.777813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10499.582593] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10508.960860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10510.205717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10510.268267] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10516.474620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10525.132353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10528.658593] Lustre: 281211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10534.407760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10534.459254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10549.001872] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:09:20 (1789499360) [10565.603110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10565.613629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.626517] Lustre: Skipped 1 previous similar message [10565.633605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10565.638162] Lustre: Skipped 1 previous similar message [10571.754959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10571.763238] Lustre: Skipped 7 previous similar messages [10576.883753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10576.891491] Lustre: Skipped 2 previous similar messages [10578.399424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10578.550598] Lustre: server umount lustre-MDT0000 complete [10582.005340] LustreError: 278212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.034816] LustreError: 278212:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [10582.531944] LustreError: 278173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789499396 with bad export cookie 3453999728380255922 [10582.552991] LustreError: 278173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10582.556162] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10587.120159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10589.255844] Lustre: server umount lustre-MDT0001 complete [10599.787258] Lustre: server umount lustre-OST0000 complete [10603.254946] Lustre: server umount lustre-OST0001 complete [10610.213620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [10620.067642] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10667.575257] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10680.155740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10680.591660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10680.655854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10680.832090] Lustre: lustre-MDT0000: new disk, initializing [10681.094265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10686.751994] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10700.657401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10700.884512] Lustre: 285740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10701.321565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10707.106532] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10712.717546] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10719.203273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10719.496732] Lustre: 287373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10721.424426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10726.237380] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10738.009429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10738.192915] Lustre: 288244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10738.358923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10738.369468] Lustre: Skipped 6 previous similar messages [10740.201604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10740.212030] Lustre: Skipped 1 previous similar message [10740.342444] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10746.674104] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10759.667342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10782.751622] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:13:13 (1789499593) [10786.286016] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10786.291322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10786.312423] Lustre: Skipped 8 previous similar messages [10786.315420] Lustre: Skipped 3 previous similar messages [10790.831525] Lustre: server umount lustre-MDT0000 complete [10796.704252] LustreError: 287372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789499610 with bad export cookie 3453999728380263839 [10796.728251] LustreError: 287372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10796.747059] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10797.330974] Lustre: server umount lustre-MDT0001 complete [10813.151374] Lustre: server umount lustre-OST0000 complete [10827.971274] Lustre: server umount lustre-OST0001 complete [10837.318418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.426194] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10843.120125] LustreError: 290495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.147536] LustreError: 290495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10852.303427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10856.982930] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10864.784416] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10871.197363] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10877.619136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.796126] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10892.785164] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10914.046874] loop1: detected capacity change from 0 to 204800000 [10914.361581] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10914.377407] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10914.396893] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10920.082108] loop1: detected capacity change from 0 to 204800000 [10926.201739] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.349871] LustreError: 290484:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs8762: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10926.360121] LustreError: 290484:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs8762 [10926.363894] LustreError: 290484:0:(mgs_handler.c:517:mgs_target_reg()) fs8762-MDT0001: Error checking target index: rc = -13 [10926.370579] LustreError: fs8762-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10926.377783] LustreError: 294215:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10926.382554] LustreError: 294215:0:(tgt_mount.c:2128:server_put_super()) no obd fs8762-MDT0001 [10926.386468] LustreError: 294215:0:(tgt_mount.c:116:server_deregister_mount()) fs8762-MDT0001 not registered [10926.417788] Lustre: server umount fs8762-MDT0001 complete [10926.426369] LustreError: 294215:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10936.018635] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.324218] Lustre: Setting parameter fs8762-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs8762-MDT0001 [10936.360034] Lustre: Skipped 1 previous similar message [10936.493190] Lustre: srv-fs8762-MDT0001: No data found on store. Initialize space. [10936.504761] Lustre: Skipped 4 previous similar messages [10936.718576] Lustre: fs8762-MDT0001: new disk, initializing [10936.727138] Lustre: Skipped 3 previous similar messages [10936.960583] Lustre: srv-fs8762-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10936.966383] Lustre: 294682:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8762-MDT0001-mdtlov: can't create new llog: rc = -115 [10936.980146] LustreError: 294682:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8762-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10941.280875] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10949.887298] Lustre: server umount fs8762-MDT0001 complete [10964.163648] loop1: detected capacity change from 0 to 204800000 [10964.466573] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10970.095886] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.518654] Lustre: 296248:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8762-MDT0001-mdtlov: can't create new llog: rc = -115 [10970.525858] LustreError: 296248:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8762-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10975.332049] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10980.465974] Lustre: server umount fs8762-MDT0001 complete [10990.706737] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [10990.713464] Lustre: Skipped 1 previous similar message [10991.076531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10991.084512] LustreError: Skipped 1 previous similar message [10991.088656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.462911] Lustre: server umount lustre-MDT0000 complete [11004.958982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11005.156188] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11010.278208] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11011.044617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11017.561028] loop1: detected capacity change from 0 to 204800000 [11023.528736] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11024.035417] Lustre: 298698:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8762-MDT0001-mdtlov: can't create new llog: rc = -115 [11024.064073] LustreError: 298698:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8762-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11028.897778] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11033.812304] Lustre: server umount fs8762-MDT0001 complete [11056.608958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11056.617158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11056.629786] LustreError: Skipped 1 previous similar message [11056.645080] Lustre: Skipped 9 previous similar messages [11068.383310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11068.511148] Lustre: server umount lustre-OST0000 complete [11086.303100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11086.540418] Lustre: server umount lustre-MDT0000 complete [11090.126043] LustreError: 290476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789499903 with bad export cookie 3453999728380266520 [11090.129509] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11090.145145] LustreError: 290476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11090.548755] Lustre: server umount lustre-MDT0001 complete [11100.600441] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11103.515468] Key type lgssc unregistered [11103.873776] LNet: 301042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11103.894041] LNetError: 301042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11103.926310] LNet: Removed LNI 192.168.204.126@tcp [11104.865163] Key type .llcrypt unregistered [11104.867794] Key type ._llcrypt unregistered [11125.901481] Key type ._llcrypt registered [11125.903867] Key type .llcrypt registered [11126.004423] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11127.099292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11127.199375] alg: No test for adler32 (adler32-zlib) [11128.316642] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11128.555319] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11130.224093] Key type lgssc registered [11131.509203] Lustre: Echo OBD driver; http://www.lustre.org/ [11138.907297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11138.938889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.638620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11145.230667] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11150.819686] LustreError: 302312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11150.869638] LustreError: 302312:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11154.877085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11155.476568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11160.720632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11169.013644] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11174.637520] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11181.200099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11181.582420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11185.651381] LustreError: 304143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.672676] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11190.754647] LustreError: 304141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11194.852327] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11195.873743] LustreError: 304373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11200.994121] LustreError: 304143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11201.021646] LustreError: 304143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11211.251777] LustreError: 304373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11211.280163] LustreError: 304373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11218.328291] loop1: detected capacity change from 0 to 204800000 [11218.542921] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11218.564736] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11218.603785] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11224.045231] loop1: detected capacity change from 0 to 204800000 [11228.273640] LustreError: 304144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11228.288098] LustreError: 304144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11229.689381] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11229.875477] Lustre: fs31443-OST0000: new disk, initializing [11229.879459] Lustre: srv-fs31443-OST0000: No data found on store. Initialize space. [11229.885745] Lustre: 306048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11229.969518] Lustre: fs31443-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11236.401839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11245.088182] Lustre: server umount fs31443-OST0000 complete [11256.415732] loop1: detected capacity change from 0 to 204800000 [11261.304427] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11261.563883] Lustre: fs31443-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11262.436640] LustreError: 305418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.453825] LustreError: 305418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [11267.569975] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11267.927778] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11275.901664] loop2: detected capacity change from 0 to 204800000 [11276.170391] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11276.190428] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11276.203598] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11281.255964] loop2: detected capacity change from 0 to 204800000 [11282.934829] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11286.835749] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11286.958719] LustreError: fs31443-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11286.968053] LustreError: fs31443-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11286.978612] LustreError: 309087:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11286.989624] LustreError: 309087:0:(tgt_mount.c:2128:server_put_super()) no obd fs31443-OST0001 [11286.995640] LustreError: 309087:0:(tgt_mount.c:116:server_deregister_mount()) fs31443-OST0001 not registered [11287.033606] Lustre: server umount fs31443-OST0001 complete [11287.037326] LustreError: 309087:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11297.320533] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11297.435965] Lustre: fs31443-OST0001: new disk, initializing [11297.439345] Lustre: srv-fs31443-OST0001: No data found on store. Initialize space. [11297.445763] Lustre: 309637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11297.515699] Lustre: fs31443-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11303.659367] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11312.736529] Lustre: server umount fs31443-OST0000 complete [11323.181920] Lustre: server umount fs31443-OST0001 complete [11329.005375] LustreError: 304143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11329.029824] LustreError: 304143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11335.616720] loop1: detected capacity change from 0 to 204800000 [11341.124195] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11341.300961] Lustre: fs31443-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11347.897392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11354.678268] loop2: detected capacity change from 0 to 204800000 [11359.874754] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11366.385890] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11375.511925] Lustre: server umount fs31443-OST0000 complete [11385.439555] Lustre: server umount fs31443-OST0001 complete [11400.677242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11400.682542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11400.683754] LustreError: Skipped 1 previous similar message [11400.687769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11400.699132] Lustre: Skipped 1 previous similar message [11400.721952] Lustre: Skipped 1 previous similar message [11404.048687] Lustre: server umount lustre-OST0000 complete [11410.916691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.935205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11410.946269] Lustre: Skipped 1 previous similar message [11413.916119] Lustre: server umount lustre-MDT0000 complete [11418.276447] LustreError: 302291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789500232 with bad export cookie 2326765637472662101 [11418.278552] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11418.286051] LustreError: 302291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11418.880815] Lustre: server umount lustre-MDT0001 complete [11429.611608] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11432.316124] Key type lgssc unregistered [11432.631146] LNet: 315561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11432.646078] LNetError: 315561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11432.664462] LNet: Removed LNI 192.168.204.126@tcp [11433.452142] Key type .llcrypt unregistered [11433.456816] Key type ._llcrypt unregistered [11467.324493] Key type ._llcrypt registered [11467.326411] Key type .llcrypt registered [11467.458496] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [11484.216993] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11485.078176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11485.149666] alg: No test for adler32 (adler32-zlib) [11486.232807] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11486.440640] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11488.087436] Key type lgssc registered [11489.228438] Lustre: Echo OBD driver; http://www.lustre.org/ [11530.860768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11530.902531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11532.265089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11532.352140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11532.489541] Lustre: lustre-MDT0000: new disk, initializing [11532.605738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11532.629497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11537.482362] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11549.419157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11549.640132] Lustre: 319420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11549.739774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11549.744443] Lustre: Skipped 1 previous similar message [11549.866129] Lustre: lustre-MDT0001: new disk, initializing [11550.036852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11550.080578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11550.102776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11555.385772] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11565.323048] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11571.464904] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11578.531685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11578.821645] Lustre: lustre-OST0000: new disk, initializing [11578.825122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11578.829295] Lustre: 321388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11578.916632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11584.617244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11584.634529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11584.828566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11585.488165] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11597.472344] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11599.361420] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479 [11600.346559] Lustre: Unmounted lustre-client [11610.595227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11610.595354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11610.605531] LustreError: Skipped 1 previous similar message [11610.620707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11610.624251] Lustre: Skipped 1 previous similar message [11614.689680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11616.216717] Lustre: server umount lustre-OST0000 complete [11619.809253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11619.812750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11619.826333] Lustre: Skipped 2 previous similar messages [11619.845270] Lustre: Skipped 2 previous similar messages [11624.946213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11624.954415] Lustre: Skipped 1 previous similar message [11625.689171] Lustre: server umount lustre-MDT0000 complete [11629.438507] LustreError: 319411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789500443 with bad export cookie 10751030909695139088 [11629.440417] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11629.446566] LustreError: 319411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11629.785385] Lustre: server umount lustre-MDT0001 complete [11640.962167] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11643.460749] Key type lgssc unregistered [11656.031420] LNet: 323550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11656.046805] LNetError: 323550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11728.808213] LNet: Removed LNI 192.168.204.126@tcp [11729.670596] Key type .llcrypt unregistered [11729.676627] Key type ._llcrypt unregistered [11748.630259] Key type ._llcrypt registered [11748.634133] Key type .llcrypt registered [11748.733325] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [11763.613510] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11764.652823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11764.673585] alg: No test for adler32 (adler32-zlib) [11765.774765] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11766.004746] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11767.695225] Key type lgssc registered [11768.882283] Lustre: Echo OBD driver; http://www.lustre.org/ [11796.841123] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:30:08 (1789500608) [11803.121152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11803.161905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11804.504957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11804.541630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11804.605730] Lustre: lustre-MDT0000: new disk, initializing [11804.710631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11804.721957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11808.967726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11820.214309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11820.346129] Lustre: 326769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11820.454858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11820.458495] Lustre: Skipped 1 previous similar message [11820.531131] Lustre: lustre-MDT0001: new disk, initializing [11820.636761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11820.668544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11820.679078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11825.099917] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11834.161428] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11840.022844] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11847.875835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11848.291299] Lustre: lustre-OST0000: new disk, initializing [11848.300724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11848.313045] Lustre: 328735:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11848.464918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11855.953033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11855.969318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11856.162157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11856.795806] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11868.629226] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11887.072020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11887.077279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.086853] LustreError: Skipped 1 previous similar message [11887.088277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11887.114647] Lustre: Skipped 1 previous similar message [11889.639323] Lustre: server umount lustre-OST0000 complete [11897.312491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11897.323212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11897.327408] Lustre: Skipped 2 previous similar messages [11899.658784] Lustre: server umount lustre-MDT0000 complete [11903.090331] LustreError: 326760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789500716 with bad export cookie 8871477898201270879 [11903.099170] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11903.110436] LustreError: 326760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11903.717916] Lustre: server umount lustre-MDT0001 complete [11913.916623] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11916.498980] Key type lgssc unregistered [11916.791143] LNet: 330644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11916.807088] LNetError: 330644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11916.834987] LNet: Removed LNI 192.168.204.126@tcp [11917.605802] Key type .llcrypt unregistered [11917.612869] Key type ._llcrypt unregistered [11938.278396] Key type ._llcrypt registered [11938.282931] Key type .llcrypt registered [11938.410601] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11939.719747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11939.797651] alg: No test for adler32 (adler32-zlib) [11940.810910] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11941.028757] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11942.743242] Key type lgssc registered [11943.865627] Lustre: Echo OBD driver; http://www.lustre.org/ [11951.944747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11951.967837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.432894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11957.870427] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11963.873130] LustreError: 331918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11963.902259] LustreError: 331918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11966.708757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11967.055705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11971.163085] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11977.762556] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11983.123363] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11990.168665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11990.493891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11996.038212] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12002.766528] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12013.675884] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12015.220649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12019.175325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.180071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12019.185095] Lustre: Skipped 1 previous similar message [12021.357981] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [12021.362420] Lustre: Skipped 2 previous similar messages [12024.290751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12024.301181] Lustre: Skipped 2 previous similar messages [12025.018866] Lustre: server umount lustre-MDT0000 complete [12026.445729] LustreError: 331914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.802477] LustreError: 335035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789500842 with bad export cookie 2698751550146576606 [12028.805317] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.815853] LustreError: 335035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12029.411445] LustreError: 331914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12029.415798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.446951] Lustre: Skipped 1 previous similar message [12029.452806] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12031.564517] LustreError: 331915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.685313] LustreError: 331913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.689616] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [12036.708279] LustreError: 331913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12036.734375] Lustre: Skipped 2 previous similar messages [12043.232594] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12043.462346] Lustre: server umount lustre-MDT0001 complete [12047.376285] Lustre: server umount lustre-OST0000 complete [12057.021534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12061.920685] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12064.925348] Lustre: 336100:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.126@tcp [12067.954271] Lustre: server umount MGS complete [12076.967319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.352509] LustreError: 336672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.375637] LustreError: 336672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12077.421443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12081.277634] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12089.839558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.155808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12094.134638] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12103.107701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12103.597787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12109.754284] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12116.827766] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12118.388325] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12123.936760] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12129.787430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [12135.396709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12135.404275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.417738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12135.427423] Lustre: Skipped 3 previous similar messages [12136.522495] Lustre: server umount lustre-OST0000 complete [12141.540258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12141.550779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12153.826337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12153.841365] Lustre: Skipped 6 previous similar messages [12154.335296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12154.581049] Lustre: server umount lustre-MDT0000 complete [12158.525273] LustreError: 336652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789500972 with bad export cookie 2698751550146577446 [12158.530391] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12158.534383] LustreError: 336652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12158.943585] Lustre: server umount lustre-MDT0001 complete [12169.643319] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12172.475988] Key type lgssc unregistered [12172.767840] LNet: 340108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12172.774606] LNetError: 340108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12172.792590] LNet: Removed LNI 192.168.204.126@tcp [12173.658761] Key type .llcrypt unregistered [12173.660597] Key type ._llcrypt unregistered [12192.922752] Key type ._llcrypt registered [12192.925778] Key type .llcrypt registered [12193.087142] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12210.408543] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12211.648728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12211.965345] alg: No test for adler32 (adler32-zlib) [12213.406316] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12213.911654] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12215.655267] Key type lgssc registered [12217.337177] Lustre: Echo OBD driver; http://www.lustre.org/ [12256.579777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12256.612904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.035954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12258.082163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12258.209909] Lustre: lustre-MDT0000: new disk, initializing [12258.356472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12258.381295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12264.528992] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12277.589631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.761581] Lustre: 343274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12277.802137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12277.804797] Lustre: Skipped 1 previous similar message [12277.857480] Lustre: lustre-MDT0001: new disk, initializing [12277.957793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12278.001386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12278.013639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12282.903657] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12293.957740] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12300.652398] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12309.271573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12309.555315] Lustre: lustre-OST0000: new disk, initializing [12309.558648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12309.569480] Lustre: 345241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12309.697585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12316.685095] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12316.779400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12316.786363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12316.927808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12326.999075] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12332.647364] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12332.966820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12338.696110] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12338.921276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12342.754823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12342.759939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12342.775395] LustreError: Skipped 1 previous similar message [12342.783302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12342.794704] Lustre: Skipped 1 previous similar message [12342.817710] Lustre: Skipped 1 previous similar message [12347.071818] Lustre: server umount lustre-OST0000 complete [12352.993131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12352.998410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12353.011764] Lustre: Skipped 1 previous similar message [12357.591476] Lustre: server umount lustre-MDT0000 complete [12361.840629] LustreError: 345240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789501175 with bad export cookie 3217709419486723490 [12361.856113] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12361.856694] LustreError: 345240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12362.302783] Lustre: server umount lustre-MDT0001 complete [12372.094797] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:39:43 (1789501183) [12391.187768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12402.118807] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12438.161689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12438.535154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12438.603925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12438.700676] Lustre: lustre-MDT0000: new disk, initializing [12438.833203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12438.873822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12444.113907] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12455.113068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12455.308667] Lustre: 350001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12455.388156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12455.398134] Lustre: Skipped 1 previous similar message [12455.490967] Lustre: lustre-MDT0001: new disk, initializing [12455.569433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12455.625569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12455.636486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12460.174131] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12471.129479] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12477.452469] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12486.309971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12486.610541] Lustre: lustre-OST0000: new disk, initializing [12486.614858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12486.622851] Lustre: 351971:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12486.709512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12492.863870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12492.876455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12493.036863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12493.124844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12504.617980] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12549.601466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12549.602863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12549.624350] LustreError: Skipped 1 previous similar message [12549.627372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12549.627384] Lustre: Skipped 1 previous similar message [12549.670862] Lustre: Skipped 1 previous similar message [12554.730245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12554.744338] Lustre: Skipped 2 previous similar messages [12559.856727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12559.865427] Lustre: Skipped 1 previous similar message [12561.377196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12561.745474] Lustre: server umount lustre-OST0000 complete [12568.544984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12568.561744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12568.585934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12572.532338] Lustre: server umount lustre-MDT0000 complete [12575.202027] LustreError: 352791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12575.227265] LustreError: 352791:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12577.103392] LustreError: 349994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789501390 with bad export cookie 3217709419486724372 [12577.110326] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12577.111384] LustreError: 349994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12577.932226] Lustre: server umount lustre-MDT0001 complete [12593.787584] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12598.646940] Key type lgssc unregistered [12599.020897] LNet: 354150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12599.032786] LNetError: 354150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12599.062210] LNet: Removed LNI 192.168.204.126@tcp [12600.169281] Key type .llcrypt unregistered [12600.173933] Key type ._llcrypt unregistered [12708.847978] Key type ._llcrypt registered [12708.850186] Key type .llcrypt registered [12708.957578] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12709.274477] Key type .llcrypt unregistered [12709.276739] Key type ._llcrypt unregistered [12788.350083] Key type ._llcrypt registered [12788.352766] Key type .llcrypt registered [12788.502345] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12788.773310] Key type .llcrypt unregistered [12788.775965] Key type ._llcrypt unregistered [12818.432691] Key type ._llcrypt registered [12818.434430] Key type .llcrypt registered [12818.561918] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12818.991419] Key type .llcrypt unregistered [12818.995061] Key type ._llcrypt unregistered [12853.079233] Key type ._llcrypt registered [12853.082123] Key type .llcrypt registered [12853.175694] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12853.453335] Key type .llcrypt unregistered [12853.461210] Key type ._llcrypt unregistered [12931.975982] Key type ._llcrypt registered [12931.978150] Key type .llcrypt registered [12932.074859] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12932.380498] Key type .llcrypt unregistered [12932.382675] Key type ._llcrypt unregistered [12960.816908] Key type ._llcrypt registered [12960.820296] Key type .llcrypt registered [12960.958467] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12961.234596] Key type .llcrypt unregistered [12961.238820] Key type ._llcrypt unregistered [12994.305269] Key type ._llcrypt registered [12994.313276] Key type .llcrypt registered [12994.552314] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12994.922925] Key type .llcrypt unregistered [12994.924763] Key type ._llcrypt unregistered [13030.866626] Key type ._llcrypt registered [13030.874201] Key type .llcrypt registered [13031.056123] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13031.541195] Key type .llcrypt unregistered [13031.544463] Key type ._llcrypt unregistered [13064.520337] Key type ._llcrypt registered [13064.523172] Key type .llcrypt registered [13064.614551] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13064.900572] Key type .llcrypt unregistered [13064.903349] Key type ._llcrypt unregistered [13095.808363] Key type ._llcrypt registered [13095.811865] Key type .llcrypt registered [13095.975797] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13096.438473] Key type .llcrypt unregistered [13096.441863] Key type ._llcrypt unregistered [13159.694402] Key type ._llcrypt registered [13159.695814] Key type .llcrypt registered [13159.851172] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13160.212845] Key type .llcrypt unregistered [13160.217316] Key type ._llcrypt unregistered [13192.323524] Key type ._llcrypt registered [13192.328404] Key type .llcrypt registered [13192.453527] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13192.859211] Key type .llcrypt unregistered [13192.861633] Key type ._llcrypt unregistered [13221.878894] Key type ._llcrypt registered [13221.881167] Key type .llcrypt registered [13221.983708] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13222.350496] Key type .llcrypt unregistered [13222.354291] Key type ._llcrypt unregistered [13271.612776] Key type ._llcrypt registered [13271.615268] Key type .llcrypt registered [13271.729760] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [13286.522228] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13287.482730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13287.611594] alg: No test for adler32 (adler32-zlib) [13288.729301] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [13289.024368] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13290.719269] Key type lgssc registered [13291.711239] Lustre: Echo OBD driver; http://www.lustre.org/ [13326.114191] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13326.141818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13327.444873] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13327.480438] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13327.586439] Lustre: tst250fs-MDT0000: new disk, initializing [13327.694477] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13327.712664] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13333.200682] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13346.727739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13346.851485] Lustre: 374416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13346.939507] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13346.945066] Lustre: Skipped 1 previous similar message [13347.037756] Lustre: tst250fs-MDT0001: new disk, initializing [13347.160402] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13347.187224] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13347.211948] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13352.542073] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13359.352369] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13367.690263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13368.183366] Lustre: tst250fs-OST0000: new disk, initializing [13368.189500] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13368.199228] Lustre: 376050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13368.342340] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13374.088127] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13374.113977] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13374.271653] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13376.024934] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13386.704509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13386.880579] Lustre: tst250fs-OST0001: new disk, initializing [13386.884279] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13386.890242] Lustre: 376920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13386.977943] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13392.503320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13396.543677] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13396.554833] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13396.631967] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13397.679325] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13400.066549] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13400.072620] Lustre: Skipped 2 previous similar messages [13418.977692] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13418.985168] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13419.021996] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13422.566777] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13422.574188] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13422.586219] Lustre: Skipped 1 previous similar message [13422.591169] Lustre: Skipped 1 previous similar message [13425.060841] Lustre: server umount tst250fs-MDT0000 complete [13429.280344] LustreError: 377532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789502243 with bad export cookie 11938032906516357406 [13429.288245] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13429.299439] LustreError: 377532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13429.747549] Lustre: server umount tst250fs-MDT0001 complete [13443.615742] Lustre: server umount tst250fs-OST0000 complete [13456.517581] Lustre: server umount tst250fs-OST0001 complete [13476.403901] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [13485.408528] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13519.487306] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:58:50 (1789502330) [13521.899232] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13543.267198] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [13554.932042] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13591.067804] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13260 sec ========= 16:00:02 (1789502402) [13593.279244] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:00:04 (1789502404) === [13596.794107] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:00:07 (1789502407) === [13636.477247] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13638.938059] Key type lgssc unregistered [13639.171047] LNet: 386050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13639.176355] LNetError: 386050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13639.203923] LNet: Removed LNI 192.168.204.126@tcp [13640.048242] Key type .llcrypt unregistered [13640.052833] Key type ._llcrypt unregistered