[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501006697 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003181] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007778] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008036] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010203] LSM: Security Framework initializing [ 0.011093] Yama: becoming mindful. [ 0.012069] SELinux: Initializing. [ 0.013106] *** VALIDATE selinux *** [ 0.021507] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026247] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027221] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028142] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029159] *** VALIDATE tmpfs *** [ 0.031448] *** VALIDATE proc *** [ 0.033083] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.036081] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037205] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039042] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.043764] debug: unmapping init [mem 0xffffffffa1259000-0xffffffffa1260fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046805] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047036] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.055326] rcu: Hierarchical SRCU implementation. [ 0.057810] smp: Bringing up secondary CPUs ... [ 0.058712] x86: Booting SMP configuration: [ 0.059038] .... node #0, CPUs: #1 #2 #3 [ 0.062727] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239343] node 0 deferred pages initialised in 172ms [ 0.242469] devtmpfs: initialized [ 0.244305] x86/mm: Memory block size: 128MB [ 0.247880] gcov: version magic: 0x41383552 [ 0.250245] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.253125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.255315] pinctrl core: initialized pinctrl subsystem [ 0.258178] [ 0.258714] ************************************************************* [ 0.260012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262015] ** ** [ 0.264013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.266013] ** ** [ 0.268013] ** This means that this kernel is built to expose internal ** [ 0.270013] ** IOMMU data structures, which may compromise security on ** [ 0.272014] ** your system. ** [ 0.274013] ** ** [ 0.277024] ** If you see this message and you are not debugging the ** [ 0.279013] ** kernel, report this immediately to your vendor! ** [ 0.281015] ** ** [ 0.283018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285018] ************************************************************* [ 0.288271] NET: Registered protocol family 16 [ 0.289399] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.291069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297146] cpuidle: using governor menu [ 0.298710] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.300453] PCI: Using configuration type 1 for base access [ 0.303141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312076] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.318089] cryptd: max_cpu_qlen set to 1000 [ 0.320345] ACPI: Added _OSI(Module Device) [ 0.321030] ACPI: Added _OSI(Processor Device) [ 0.322041] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.328110] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335200] ACPI: Interpreter enabled [ 0.337062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338019] ACPI: Using IOAPIC for interrupt routing [ 0.340163] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.343449] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.353688] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.358019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.361094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.366450] acpiphp: Slot [2] registered [ 0.368143] acpiphp: Slot [5] registered [ 0.370121] acpiphp: Slot [6] registered [ 0.371111] acpiphp: Slot [7] registered [ 0.372140] acpiphp: Slot [8] registered [ 0.374131] acpiphp: Slot [9] registered [ 0.375131] acpiphp: Slot [10] registered [ 0.376131] acpiphp: Slot [3] registered [ 0.378103] acpiphp: Slot [4] registered [ 0.379092] acpiphp: Slot [11] registered [ 0.381128] acpiphp: Slot [12] registered [ 0.382104] acpiphp: Slot [13] registered [ 0.383116] acpiphp: Slot [14] registered [ 0.384114] acpiphp: Slot [15] registered [ 0.386088] acpiphp: Slot [16] registered [ 0.387079] acpiphp: Slot [17] registered [ 0.388105] acpiphp: Slot [18] registered [ 0.389129] acpiphp: Slot [19] registered [ 0.391136] acpiphp: Slot [20] registered [ 0.392114] acpiphp: Slot [21] registered [ 0.394096] acpiphp: Slot [22] registered [ 0.395110] acpiphp: Slot [23] registered [ 0.397125] acpiphp: Slot [24] registered [ 0.398098] acpiphp: Slot [25] registered [ 0.400125] acpiphp: Slot [26] registered [ 0.401068] acpiphp: Slot [27] registered [ 0.402103] acpiphp: Slot [28] registered [ 0.403101] acpiphp: Slot [29] registered [ 0.405100] acpiphp: Slot [30] registered [ 0.406099] acpiphp: Slot [31] registered [ 0.407073] PCI host bridge to bus 0000:00 [ 0.409019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.411024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.413022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.415024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.417025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.420029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.422196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.425051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.428380] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.437761] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.442390] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.444018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.448019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.450613] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.453759] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.456044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.459838] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.463966] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.474766] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.478795] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.484896] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.494029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.501020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.521023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.541186] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.551022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.561022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.586020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596138] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.604020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.612023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.638017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.657524] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.665017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.674026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.695211] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.709000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.718021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.725039] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.752021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.761965] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.771021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.782021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.817023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.836878] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.839379] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.841345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.843438] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.846500] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.854012] iommu: Default domain type: Passthrough [ 0.855612] SCSI subsystem initialized [ 0.856295] ACPI: bus type USB registered [ 0.857173] usbcore: registered new interface driver usbfs [ 0.859123] usbcore: registered new interface driver hub [ 0.860114] usbcore: registered new device driver usb [ 0.861204] pps_core: LinuxPPS API ver. 1 registered [ 0.863012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.865101] PTP clock support registered [ 0.867105] EDAC MC: Ver: 3.0.0 [ 0.868494] PCI: Using ACPI for IRQ routing [ 0.870911] NetLabel: Initializing [ 0.872010] NetLabel: domain hash size = 128 [ 0.873011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.875092] NetLabel: unlabeled traffic allowed by default [ 0.877144] vgaarb: loaded [ 0.879356] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.881021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.891005] clocksource: Switched to clocksource kvm-clock [ 1.004635] VFS: Disk quotas dquot_6.6.0 [ 1.006376] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.008938] *** VALIDATE ramfs *** [ 1.010269] *** VALIDATE hugetlbfs *** [ 1.012044] pnp: PnP ACPI init [ 1.014785] pnp: PnP ACPI: found 6 devices [ 1.034080] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.036450] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.038147] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.040136] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.042226] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.044364] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.047174] NET: Registered protocol family 2 [ 1.049665] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.054379] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.057378] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.061837] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.065192] TCP: Hash tables configured (established 65536 bind 65536) [ 1.067220] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.069903] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.072722] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.075200] NET: Registered protocol family 1 [ 1.077750] RPC: Registered named UNIX socket transport module. [ 1.079030] RPC: Registered udp transport module. [ 1.080625] RPC: Registered tcp transport module. [ 1.082203] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.084437] NET: Registered protocol family 44 [ 1.086488] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.089594] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.092352] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.094959] PCI: CLS 0 bytes, default 64 [ 1.096684] Unpacking initramfs... [ 2.541409] debug: unmapping init [mem 0xffff97c9bcc54000-0xffff97c9bffbffff] [ 2.545340] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.547630] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.550438] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.055508] Initialise system trusted keyrings [ 3.057312] Key type blacklist registered [ 3.059224] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.066719] zbud: loaded [ 3.069519] *** VALIDATE nfs *** [ 3.070419] *** VALIDATE nfs4 *** [ 3.071439] pstore: using deflate compression [ 3.074516] Platform Keyring initialized [ 3.181269] NET: Registered protocol family 38 [ 3.182769] Key type asymmetric registered [ 3.183984] Asymmetric key parser 'x509' registered [ 3.185442] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.188242] io scheduler mq-deadline registered [ 3.189597] io scheduler kyber registered [ 3.191265] io scheduler bfq registered [ 3.192812] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.194963] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.197558] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.200157] ACPI: Power Button [PWRF] [ 3.206263] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.212833] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.241193] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.252106] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.273813] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.303059] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.333184] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.339127] Non-volatile memory driver v1.3 [ 3.340704] Linux agpgart interface v0.103 [ 3.372537] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.375068] vda: detected capacity change from 0 to 74702848 [ 3.389734] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.392860] vdb: detected capacity change from 0 to 1073741824 [ 3.409474] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.412424] vdc: detected capacity change from 0 to 2621440000 [ 3.429405] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.433027] vdd: detected capacity change from 0 to 2621440000 [ 3.447558] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.449862] vde: detected capacity change from 0 to 4294967296 [ 3.463302] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.466444] vdf: detected capacity change from 0 to 4294967296 [ 3.477760] libphy: Fixed MDIO Bus: probed [ 3.483449] usbcore: registered new interface driver usbserial_generic [ 3.485996] usbserial: USB Serial support registered for generic [ 3.488296] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.492416] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.493955] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.496362] mousedev: PS/2 mouse device common for all mice [ 3.499287] rtc_cmos 00:05: RTC can wake from S4 [ 3.501118] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.503516] rtc_cmos 00:05: registered as rtc0 [ 3.505693] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.506395] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.510573] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.512140] intel_pstate: CPU model not supported [ 3.518755] hid: raw HID events driver (C) Jiri Kosina [ 3.520910] usbcore: registered new interface driver usbhid [ 3.522861] usbhid: USB HID core driver [ 3.524373] drop_monitor: Initializing network drop monitor service [ 3.526694] Initializing XFRM netlink socket [ 3.528644] NET: Registered protocol family 10 [ 3.532105] Segment Routing with IPv6 [ 3.533845] NET: Registered protocol family 17 [ 3.535859] mpls_gso: MPLS GSO support [ 3.541563] RAS: Correctable Errors collector initialized. [ 3.543979] AVX version of gcm_enc/dec engaged. [ 3.545810] AES CTR mode by8 optimization enabled [ 3.628067] sched_clock: Marking stable (3628031360, 0)->(4584898612, -956867252) [ 3.631430] registered taskstats version 1 [ 3.633241] Loading compiled-in X.509 certificates [ 3.635552] zswap: loaded using pool lzo/zbud [ 3.662387] Key type big_key registered [ 3.675601] Key type encrypted registered [ 3.677093] ima: No TPM chip found, activating TPM-bypass! [ 3.678722] ima: Allocated hash algorithm: sha1 [ 3.680247] ima: No architecture policies found [ 3.681775] evm: Initialising EVM extended attributes: [ 3.683316] evm: security.selinux [ 3.684295] evm: security.ima [ 3.685230] evm: security.capability [ 3.686358] evm: HMAC attrs: 0x1 [ 3.688636] rtc_cmos 00:05: setting system clock to 2026-08-14 22:24:07 UTC (1786746247) [ 3.694238] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 3.697019] debug: unmapping init [mem 0xffffffffa0f82000-0xffffffffa1258fff] [ 3.705100] Write protecting the kernel read-only data: 28672k [ 3.708308] debug: unmapping init [mem 0xffffffff9f603000-0xffffffff9f7fffff] [ 3.710803] debug: unmapping init [mem 0xffffffff9ff14000-0xffffffff9fffffff] [ 3.746098] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.753930] systemd[1]: Detected virtualization kvm. [ 3.755656] systemd[1]: Detected architecture x86-64. [ 3.757314] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.783716] systemd[1]: No hostname configured. [ 3.784963] systemd[1]: Set hostname to . [ 3.786482] random: systemd: uninitialized urandom read (16 bytes read) [ 3.787939] systemd[1]: Initializing machine ID from random generator. [ 3.831655] random: ln: uninitialized urandom read (6 bytes read) [ 3.927535] random: systemd: uninitialized urandom read (16 bytes read) [ 3.930303] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.935530] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.940669] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.590473] device-mapper: uevent: version 1.0.3 [ 4.592800] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target B[ 5.363302] random: fast init done asic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.391978] virtio_net virtio0 ens2: renamed from eth0 [ 5.442028] scsi host0: ata_piix [ 5.465606] scsi host1: ata_piix [ 5.467136] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.469128] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.823297] dracut-initqueue[594]: RTNETLINK answers: File exists [ 10.194967] random: crng init done [ 10.197893] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.671222] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.684917] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.933035] SELinux: Disabled at runtime. [ 11.990798] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.998978] systemd[1]: Detected virtualization kvm. [ 12.000590] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.474635] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.477938] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.482758] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.488507] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.491428] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.499094] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.516488] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ 12.555535] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.890705] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.070913] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.169304] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.278204] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.288827] EDAC sbridge: Ver: 1.1.2 [ 14.726569] Key type dns_resolver registered [ 15.010756] NFS: Registering the id_resolver key type [ 15.012450] Key type id_resolver registered [ 15.013457] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg132-server login: [ 41.616585] libcfs: loading out-of-tree module taints kernel. [ 41.635452] Key type ._llcrypt registered [ 41.637064] Key type .llcrypt registered [ 41.691946] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 50.589440] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 51.193470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.201558] alg: No test for adler32 (adler32-zlib) [ 52.242962] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 52.590145] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 54.207311] Key type lgssc registered [ 56.261539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.310164] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 83.907091] hrtimer: interrupt took 3998872 ns [ 125.761154] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 138.339816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.369673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.636485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.662176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.721908] Lustre: lustre-MDT0000: new disk, initializing [ 139.801311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.817572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.316617] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.075731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.187349] Lustre: 6491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.210534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.214278] Lustre: Skipped 1 previous similar message [ 158.288393] Lustre: lustre-MDT0001: new disk, initializing [ 158.348602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.389099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.397803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.086447] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.812526] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 178.540312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.756474] Lustre: lustre-OST0000: new disk, initializing [ 178.760356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 178.765531] Lustre: 8429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 178.854501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.199518] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.379565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.393075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.502166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.137645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.266719] Lustre: lustre-OST0001: new disk, initializing [ 201.273854] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.277887] Lustre: 9500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.337293] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.641102] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.974298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 208.981076] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.043768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.650278] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.908829] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.332638] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.849796] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing yml_node [ 245.616192] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 248.612111] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 251.445910] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 252.897122] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 18:28:15 EDT 2026 [ 271.682901] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 273.725759] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.789351] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.800200] Lustre: Skipped 1 previous similar message [ 290.800375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.820621] Lustre: Skipped 3 previous similar messages [ 295.907156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.944132] Lustre: server umount lustre-MDT0000 complete [ 301.362945] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746545 with bad export cookie 10739604674725701825 [ 301.374691] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.384115] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 306.147757] LustreError: 9787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.148462] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.174524] LustreError: 9787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 306.206176] Lustre: Skipped 2 previous similar messages [ 306.213494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.219322] Lustre: Skipped 4 previous similar messages [ 307.868927] Lustre: server umount lustre-MDT0001 complete [ 312.132172] Lustre: server umount lustre-OST0000 complete [ 316.135442] Lustre: server umount lustre-OST0001 complete [ 322.983821] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 331.504689] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 371.030663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.374262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.406098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.507314] Lustre: lustre-MDT0000: new disk, initializing [ 371.601659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.619940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 377.428821] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 388.514312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.733175] Lustre: 15053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 388.797480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 388.801828] Lustre: Skipped 1 previous similar message [ 388.976972] Lustre: lustre-MDT0001: new disk, initializing [ 389.124964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 389.185403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 389.204229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 394.342357] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 404.538491] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.255185] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 417.166904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 417.480582] Lustre: lustre-OST0000: new disk, initializing [ 417.490049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 417.497778] Lustre: 17025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 417.593878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.045580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 419.058660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 419.183662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 424.676814] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 435.010927] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 440.308514] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 440.559916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 445.077501] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 445.302470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 450.016819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.039726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.051580] Lustre: Skipped 1 previous similar message [ 450.056261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.153079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.159059] Lustre: Skipped 2 previous similar messages [ 462.303938] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 462.686100] Lustre: server umount lustre-OST0000 complete [ 470.496805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.508128] Lustre: Skipped 1 previous similar message [ 470.514777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.530027] Lustre: Skipped 2 previous similar messages [ 473.772516] Lustre: server umount lustre-MDT0000 complete [ 475.634219] LustreError: 15064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.668754] LustreError: 15064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 477.983310] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746721 with bad export cookie 10739604674725703358 [ 477.996803] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.303317] Lustre: server umount lustre-MDT0001 complete [ 484.383660] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:32:06 (1786746726) [ 505.619828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.253374] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.342952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.458879] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.700388] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 516.582372] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.505337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.699837] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 526.783498] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 534.455782] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.814635] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 543.331132] Lustre: Failing over lustre-MDT0000 [ 543.357632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.373799] Lustre: Skipped 1 previous similar message [ 543.737672] Lustre: server umount lustre-MDT0000 complete [ 546.467063] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.486164] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 563.168229] Lustre: 3623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786746790/real 1786746790] req@ffff97c903489c00 x1873539284241536/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786746806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 563.199209] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.216624] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.236141] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 564.549164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.442679] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x950abd8d01267444 [ 574.463699] Lustre: MGC192.168.201.132@tcp: Connection restored to 0@lo (at 0@lo) [ 574.824396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.832513] Lustre: Skipped 1 previous similar message [ 579.587220] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 580.072831] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 580.090764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 580.145611] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 589.583252] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 595.424593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.433471] LustreError: Skipped 1 previous similar message [ 595.435761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.451181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.455216] Lustre: Skipped 2 previous similar messages [ 599.929170] Lustre: server umount lustre-MDT0000 complete [ 601.576320] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.608190] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 603.895173] LustreError: 19883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746847 with bad export cookie 10739604674725704772 [ 603.905696] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.911374] LustreError: 19883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 604.457110] Lustre: server umount lustre-MDT0001 complete [ 613.830592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.204514] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 629.735105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.537213] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 643.786755] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 651.230532] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 653.712024] Lustre: Failing over lustre-MDT0000 [ 654.160968] Lustre: server umount lustre-MDT0000 complete [ 655.786754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.810096] Lustre: Skipped 2 previous similar messages [ 666.081585] LustreError: 24243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.110839] LustreError: 24243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 672.171491] Lustre: 3625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786746899/real 1786746899] req@ffff97ca41741180 x1873539284272000/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786746915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 672.221406] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.897333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.847563] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 688.115106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 688.121395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 688.143526] Lustre: Skipped 1 previous similar message [ 688.176935] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 688.534294] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 701.114643] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 708.576492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 708.583045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.614485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.625882] Lustre: Skipped 2 previous similar messages [ 721.377590] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 721.696237] Lustre: server umount lustre-MDT0000 complete [ 725.871437] LustreError: 23221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746969 with bad export cookie 10739604674725705710 [ 725.882578] LustreError: 23221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 725.893284] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.247364] Lustre: server umount lustre-MDT0001 complete [ 738.268663] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:36:19 (1786746979) [ 758.769351] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 769.179917] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 816.917053] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 829.068423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.402131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 829.431792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 829.524540] Lustre: lustre-MDT0000: new disk, initializing [ 829.625699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.640799] Lustre: Skipped 3 previous similar messages [ 829.656761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 835.164119] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 847.390781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.503925] Lustre: 29621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 847.880305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 853.519237] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 864.329862] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 871.514694] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 879.500705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 879.878959] Lustre: 31590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 881.303166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 881.386331] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 881.509456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 881.537149] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 888.399758] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 899.421315] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 905.836894] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 932.835261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.839062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.851348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.851359] Lustre: Skipped 6 previous similar messages [ 932.887153] Lustre: Skipped 2 previous similar messages [ 937.805192] Lustre: server umount lustre-OST0000 complete [ 948.438861] Lustre: server umount lustre-MDT0000 complete [ 952.294666] LustreError: 29629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.310345] LustreError: 29629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 952.499214] LustreError: 29612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747196 with bad export cookie 10739604674725706116 [ 952.506500] LustreError: 29612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 952.508226] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.134317] Lustre: server umount lustre-MDT0001 complete [ 968.300488] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 972.183108] Key type lgssc unregistered [ 972.700431] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.710672] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.737917] LNet: Removed LNI 192.168.201.132@tcp [ 975.438219] Key type .llcrypt unregistered [ 975.440094] Key type ._llcrypt unregistered [ 1002.095139] Key type ._llcrypt registered [ 1002.100728] Key type .llcrypt registered [ 1002.247832] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1003.842931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.896460] alg: No test for adler32 (adler32-zlib) [ 1005.100488] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 1005.449320] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1007.171775] Key type lgssc registered [ 1008.449895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1017.349373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1017.390684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.962056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.678593] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1029.090639] LustreError: 34992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.119286] LustreError: 34992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1032.975084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1033.640866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1038.810232] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1047.842481] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1055.356982] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1063.412412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.966059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.228396] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1078.318901] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1079.287075] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.641407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1106.576288] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:42:28 (1786747348) [ 1108.382488] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1110.384273] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1118.073604] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:42:39 (1786747359) [ 1128.940732] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:42:50 (1786747370) [ 1130.274414] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1140.403767] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1140.409655] Lustre: Skipped 2 previous similar messages [ 1142.689485] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:43:04 (1786747384) [ 1144.543400] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1144.552310] Lustre: Skipped 2 previous similar messages [ 1153.680178] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1153.683585] Lustre: Skipped 4 previous similar messages [ 1168.298940] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:43:30 (1786747410) [ 1171.299426] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1175.053407] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.608939] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:43:46 (1786747426) [ 1187.070798] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1187.073572] Lustre: Skipped 6 previous similar messages [ 1203.329733] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:44:05 (1786747445) [ 1221.600747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1221.612094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.626247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.637212] Lustre: Skipped 1 previous similar message [ 1221.666924] Lustre: Skipped 1 previous similar message [ 1223.708472] Lustre: server umount lustre-OST0000 complete [ 1228.256801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1228.262327] LustreError: Skipped 1 previous similar message [ 1228.265643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.275642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.841173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.843796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.265962] Lustre: server umount lustre-MDT0000 complete [ 1236.961430] LustreError: 34992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.976259] LustreError: 34992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1237.826283] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747481 with bad export cookie 7696923415186550755 [ 1237.835044] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1237.838552] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1238.118716] Lustre: server umount lustre-MDT0001 complete [ 1249.392600] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1252.133494] Key type lgssc unregistered [ 1252.413735] LNet: 42874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.423322] LNetError: 42874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.468187] LNet: Removed LNI 192.168.201.132@tcp [ 1253.233986] Key type .llcrypt unregistered [ 1253.237227] Key type ._llcrypt unregistered [ 1272.410381] Key type ._llcrypt registered [ 1272.412389] Key type .llcrypt registered [ 1272.618921] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 1290.393576] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1291.999931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1292.315786] alg: No test for adler32 (adler32-zlib) [ 1293.702508] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 1294.175691] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1295.896051] Key type lgssc registered [ 1297.794921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1336.926161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1336.956749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.418708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1338.449629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1338.541938] Lustre: lustre-MDT0000: new disk, initializing [ 1338.627225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.643270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1343.153159] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1355.063967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.220288] Lustre: 46030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1355.319983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1355.339962] Lustre: Skipped 1 previous similar message [ 1355.457729] Lustre: lustre-MDT0001: new disk, initializing [ 1355.611391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1355.711568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1355.736167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1363.140729] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1376.945538] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1384.611342] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1391.901739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.298588] Lustre: lustre-OST0000: new disk, initializing [ 1392.302358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1392.308858] Lustre: 47999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1392.408881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.378121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1398.392994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1398.492023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1399.524663] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1409.293774] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1414.449925] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1414.630899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1419.948443] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1420.188596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1424.353287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1424.356570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.357496] LustreError: Skipped 1 previous similar message [ 1424.359297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.391129] Lustre: Skipped 1 previous similar message [ 1428.276140] Lustre: server umount lustre-OST0000 complete [ 1434.598107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.612498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.622994] Lustre: Skipped 2 previous similar messages [ 1439.716081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.730062] Lustre: Skipped 1 previous similar message [ 1440.338339] Lustre: server umount lustre-MDT0000 complete [ 1445.558991] LustreError: 47998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747689 with bad export cookie 3702111503251655411 [ 1445.573492] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.575580] LustreError: 47998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1445.924112] Lustre: server umount lustre-MDT0001 complete [ 1456.749845] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:48:19 (1786747699) [ 1464.460734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.850619] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.864177] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1464.926641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.734925] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1469.927177] LustreError: 50464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.051277] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.352897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.631777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1482.121451] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1489.642308] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1497.133574] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1507.007518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.518675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.099821] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1523.639254] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1528.625823] LustreError: 52992:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1529.733966] Lustre: Modifying parameter general.timeout=1 in log params [ 1531.522873] Lustre: Modifying parameter general.timeout=3 in log params [ 1531.527988] Lustre: Skipped 1 previous similar message [ 1534.518920] Lustre: Modifying parameter general.timeout=6 in log params [ 1534.529319] Lustre: Skipped 2 previous similar messages [ 1538.741341] Lustre: Modifying parameter general.timeout=10 in log params [ 1538.747195] Lustre: Skipped 3 previous similar messages [ 1547.082168] Lustre: Modifying parameter general.timeout=18 in log params [ 1547.088120] Lustre: Skipped 7 previous similar messages [ 1563.192922] Lustre: Modifying parameter general.timeout=34 in log params [ 1563.195361] Lustre: Skipped 15 previous similar messages [ 1595.763789] Lustre: Modifying parameter general.timeout=66 in log params [ 1595.768637] Lustre: Skipped 31 previous similar messages [ 1664.272504] Lustre: Disabling parameter general.timeout= in log params [ 1664.276645] Lustre: Skipped 63 previous similar messages [ 1667.509196] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:51:49 (1786747909) [ 1670.809099] Lustre: 59937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1673.005960] Lustre: 60047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1677.057975] Lustre: 60246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1677.075882] Lustre: 60246:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1711.316986] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1711.325845] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1711.366571] LustreError: 61818:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1716.281724] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:52:37 (1786747957) [ 1731.565406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.572880] Lustre: Skipped 1 previous similar message [ 1731.578223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1731.589441] Lustre: Skipped 2 previous similar messages [ 1736.677513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.690906] Lustre: Skipped 2 previous similar messages [ 1737.784415] Lustre: server umount lustre-MDT0000 complete [ 1741.804949] LustreError: 50464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.846080] LustreError: 50464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1743.142169] LustreError: 52306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747986 with bad export cookie 3702111503251656300 [ 1743.147318] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.149682] LustreError: 52306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1743.766271] Lustre: server umount lustre-MDT0001 complete [ 1759.498436] Lustre: server umount lustre-OST0000 complete [ 1790.016621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.574462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1790.933507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.291947] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1805.386626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.540458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1805.579608] Lustre: 64357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1811.962430] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1820.373629] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1825.778677] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1833.888977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1834.232289] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1834.698304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.705909] Lustre: Skipped 1 previous similar message [ 1840.772084] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1848.970234] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.100055] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1851.109315] Lustre: Skipped 23 previous similar messages [ 1851.191414] Lustre: 66922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1859.044994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.075025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.080093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.100346] Lustre: Skipped 2 previous similar messages [ 1865.073412] Lustre: server umount lustre-OST0000 complete [ 1869.283331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.301400] Lustre: Skipped 1 previous similar message [ 1875.554577] Lustre: server umount lustre-MDT0000 complete [ 1879.499086] LustreError: 64349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748123 with bad export cookie 3702111503251659464 [ 1879.508603] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1879.514640] LustreError: 64349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1879.976776] Lustre: server umount lustre-MDT0001 complete [ 1892.718484] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1895.967925] Key type lgssc unregistered [ 1896.226964] LNet: 68134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.233782] LNetError: 68134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.248983] LNet: Removed LNI 192.168.201.132@tcp [ 1897.200765] Key type .llcrypt unregistered [ 1897.202965] Key type ._llcrypt unregistered [ 1923.481306] Key type ._llcrypt registered [ 1923.482822] Key type .llcrypt registered [ 1923.588158] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1924.972975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.139585] alg: No test for adler32 (adler32-zlib) [ 1926.624362] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 1926.956736] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1928.703190] Key type lgssc registered [ 1929.696816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.351276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1937.368375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.936877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.761652] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1949.165101] LustreError: 69550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.188668] LustreError: 69550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1953.391922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.163453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.901846] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1967.491937] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1974.836723] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.810179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1983.532586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.049884] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1998.535535] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2006.194749] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2019.298665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.305617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.333922] Lustre: Skipped 2 previous similar messages [ 2019.358684] Lustre: Skipped 2 previous similar messages [ 2023.054724] Lustre: server umount lustre-MDT0000 complete [ 2024.428026] LustreError: 69551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.469103] LustreError: 69551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2027.581535] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748271 with bad export cookie 14471277587810812810 [ 2027.588532] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2027.588745] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.366563] Lustre: server umount lustre-MDT0001 complete [ 2042.740038] Lustre: server umount lustre-OST0000 complete [ 2071.174146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.429312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2071.461812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2071.724692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.603541] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2086.138824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.260572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2086.318056] Lustre: 74424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2086.494723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2087.013883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2092.357042] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2100.977930] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2107.504668] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2115.553524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2115.765822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2116.045234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.980924] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2129.760576] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2131.822876] Lustre: 77003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2133.898441] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2141.156415] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2141.166834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.168709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.204471] Lustre: Skipped 1 previous similar message [ 2146.275345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.287076] Lustre: Skipped 2 previous similar messages [ 2147.059767] Lustre: server umount lustre-OST0000 complete [ 2153.442571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2153.456904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.481583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.538584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.897637] Lustre: server umount lustre-MDT0000 complete [ 2161.632462] LustreError: 74435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.483460] LustreError: 75169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748406 with bad export cookie 14471277587810813657 [ 2162.485996] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2162.494555] LustreError: 75169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2162.784601] Lustre: server umount lustre-MDT0001 complete [ 2177.158569] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2180.483931] Key type lgssc unregistered [ 2180.746785] LNet: 78221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.755410] LNetError: 78221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.776971] LNet: Removed LNI 192.168.201.132@tcp [ 2181.862441] Key type .llcrypt unregistered [ 2181.864202] Key type ._llcrypt unregistered [ 2210.829691] Key type ._llcrypt registered [ 2210.832596] Key type .llcrypt registered [ 2211.033076] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2212.375534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.608334] alg: No test for adler32 (adler32-zlib) [ 2213.851343] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 2214.289246] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2216.063157] Key type lgssc registered [ 2217.210322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2226.367293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2226.403397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.092250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.594438] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2237.409075] LustreError: 79648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.425272] LustreError: 79648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2242.529664] LustreError: 79649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.295677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.071399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2249.048715] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2256.879911] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2262.283702] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2269.021680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.597095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.086278] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2283.669041] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2294.292355] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2295.437335] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2296.599851] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2298.892635] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2298.897289] Lustre: Skipped 1 previous similar message [ 2303.203534] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2303.207231] Lustre: Skipped 3 previous similar messages [ 2312.130677] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2312.135679] Lustre: Skipped 8 previous similar messages [ 2328.402472] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2328.406661] Lustre: Skipped 17 previous similar messages [ 2361.178950] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2361.187441] Lustre: Skipped 38 previous similar messages [ 2397.199645] Lustre: 87751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2401.085318] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:04:02 (1786748642) [ 2402.773544] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2404.381313] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:04:06 (1786748646) [ 2410.978711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.981126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2411.022806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.001129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.008443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.009065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.029125] Lustre: Skipped 1 previous similar message [ 2415.281989] Lustre: server umount lustre-OST0000 complete [ 2422.241733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.268042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.917822] Lustre: server umount lustre-MDT0000 complete [ 2427.369309] LustreError: 84981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.393741] LustreError: 84981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2430.141540] LustreError: 80393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748673 with bad export cookie 16901893762873248405 [ 2430.142535] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.149568] LustreError: 80393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2430.445726] Lustre: server umount lustre-MDT0001 complete [ 2441.383977] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2444.603334] Key type lgssc unregistered [ 2445.035455] LNet: 89009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.042978] LNetError: 89009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.144973] LNet: Removed LNI 192.168.201.132@tcp [ 2447.131172] Key type .llcrypt unregistered [ 2447.134213] Key type ._llcrypt unregistered [ 2451.380903] Key type ._llcrypt registered [ 2451.383331] Key type .llcrypt registered [ 2451.527340] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2456.366886] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules [ 2457.710406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2457.739470] alg: No test for adler32 (adler32-zlib) [ 2458.775239] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 2459.028538] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2460.727199] Key type lgssc registered [ 2462.020851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.216334] LustreError: 90213:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2481.106604] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2484.996305] LustreError: 90213:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2491.604845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.632946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.352106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.668673] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2503.669206] LustreError: 91241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.706172] LustreError: 91241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2508.770378] LustreError: 91242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.424237] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:05:54 (1786748754) [ 2513.894634] LustreError: 91241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.562474] LustreError: 91273:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2518.567339] LustreError: 91273:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2518.572204] LustreError: 91273:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2519.182369] Lustre: server umount lustre-MDT0000 complete [ 2535.251802] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2538.542939] Key type lgssc unregistered [ 2538.848066] LNet: 92770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.860493] LNetError: 92770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2538.910119] LNet: Removed LNI 192.168.201.132@tcp [ 2539.644347] Key type .llcrypt unregistered [ 2539.651679] Key type ._llcrypt unregistered [ 2559.712587] Key type ._llcrypt registered [ 2559.714844] Key type .llcrypt registered [ 2559.828877] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2560.983607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2561.182738] alg: No test for adler32 (adler32-zlib) [ 2562.367446] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 2562.588574] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2564.215134] Key type lgssc registered [ 2565.197710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.755455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2572.790033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.209718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.029826] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2583.536036] LustreError: 93988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.549859] LustreError: 93988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2587.395032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.996228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.305122] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2600.078178] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2606.892199] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2615.335171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.287852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.977347] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2632.077295] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2682.345889] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:08:44 (1786748924) [ 2684.016977] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2685.771267] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:08:47 (1786748927) [ 2690.020581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2690.036944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.049595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.069372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.078813] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.124132] Lustre: Skipped 1 previous similar message [ 2696.302291] Lustre: server umount lustre-MDT0000 complete [ 2697.185538] LustreError: 94007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.213286] LustreError: 94007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2700.193267] LustreError: 93970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748944 with bad export cookie 6941294852052514347 [ 2700.208708] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.649382] Lustre: server umount lustre-MDT0001 complete [ 2715.289672] Lustre: server umount lustre-OST0000 complete [ 2726.867227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.173817] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.369649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.552957] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2732.525829] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.635322] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.575033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.664737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.638154] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2758.761159] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2765.379181] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2779.669399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.986127] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2779.993045] LustreError: 98271:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2780.031930] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2780.041980] LustreError: 100369:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2780.061990] LustreError: 100369:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2780.070060] LustreError: 100369:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2780.162040] Lustre: server umount lustre-OST0000 complete [ 2780.166172] LustreError: 100369:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2789.284178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.607275] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2789.619963] LustreError: 98273:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2789.653790] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2789.663403] LustreError: 100831:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2789.669453] LustreError: 100831:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2789.674651] LustreError: 100831:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2789.818202] Lustre: server umount lustre-OST0000 complete [ 2789.819984] LustreError: 100831:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2799.296806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.473802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2799.481558] Lustre: Found index 0 for lustre-OST0000, updating log [ 2799.489324] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2799.618661] Lustre: lustre-OST0000: new disk, initializing [ 2799.622209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2799.630070] Lustre: 101370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2799.899356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.080472] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2810.136853] Lustre: Failing over lustre-OST0000 [ 2810.302225] Lustre: server umount lustre-OST0000 complete [ 2814.504424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.514137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2814.516496] Lustre: Skipped 2 previous similar messages [ 2819.555524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.573886] Lustre: Skipped 1 previous similar message [ 2821.054653] Lustre: server umount lustre-MDT0000 complete [ 2824.677337] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.692808] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2824.932070] LustreError: 101369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749068 with bad export cookie 6941294852052517091 [ 2824.937256] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.940531] LustreError: 101369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2825.251989] Lustre: server umount lustre-MDT0001 complete [ 2837.451300] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:11:19 (1786749079) [ 2853.989737] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2867.651541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.321866] LustreError: 103715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.593281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.894487] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2885.470518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.666419] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2898.204454] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2906.779167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.429711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.435712] Lustre: Skipped 1 previous similar message [ 2914.904635] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2924.005850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.092208] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2924.168575] Lustre: lustre-OST0001: new disk, initializing [ 2924.180341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2924.192098] Lustre: 106033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2930.595687] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2931.815608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2931.833174] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2931.958566] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2938.343934] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2944.523674] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2952.678441] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.682989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2952.706074] Lustre: Skipped 3 previous similar messages [ 2952.714443] Lustre: Skipped 1 previous similar message [ 2955.117927] Lustre: server umount lustre-MDT0001 complete [ 2957.796679] LustreError: 103715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.808301] LustreError: 103715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2964.478778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.633587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2964.641351] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2964.704473] Lustre: 103705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2965.858717] LustreError: 107557:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2965.861363] LustreError: 107557:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2965.876785] LustreError: 103707:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2966.048828] LustreError: 107574:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.132@tcp: cfg command failed: rc = -17 [ 2966.054174] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2966.054174] [ 2970.607387] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2970.703793] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2983.834494] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:13:46 (1786749226) [ 3004.216370] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3026.977270] Lustre: 109659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3051.482814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3056.435740] Lustre: 110795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3149.291634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3149.303159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.320125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.323450] Lustre: Skipped 1 previous similar message [ 3152.574111] Lustre: server umount lustre-MDT0000 complete [ 3154.918636] LustreError: 106562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.930890] LustreError: 106562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3157.393866] LustreError: 106812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749401 with bad export cookie 6941294852052518085 [ 3157.395539] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3157.423659] LustreError: 106812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3158.337155] Lustre: server umount lustre-MDT0001 complete [ 3174.087744] Lustre: server umount lustre-OST0000 complete [ 3176.161108] Lustre: 93299:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749403/real 1786749403] req@ffff97c902361c00 x1873541916365312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3176.171348] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.175536] Lustre: Skipped 3 previous similar messages [ 3177.190178] Lustre: 93301:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749405/real 1786749405] req@ffff97ca2c728700 x1873541916365568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749421 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3178.015447] Lustre: server umount lustre-OST0001 complete [ 3186.097216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3198.917939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3208.769039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3219.664350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3244.984472] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3256.347300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.362943] Lustre: lustre-MDT0000: reset Object Index mappings [ 3256.740285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.754815] Lustre: Skipped 2 previous similar messages [ 3262.399764] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3272.651153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.691480] Lustre: lustre-MDT0001: reset Object Index mappings [ 3279.634159] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3283.081409] Lustre: 115893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3290.267644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3298.478495] LustreError: 116245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.512398] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3298.526733] LustreError: 116245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3300.002081] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3301.598434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3315.051812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3321.357205] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3321.362967] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3321.983467] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3330.295852] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3334.068873] Lustre: 117771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3386.025181] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:20:27 (1786749627) [ 3411.426122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3411.434692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.446131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.449177] Lustre: Skipped 3 previous similar messages [ 3417.841518] Lustre: server umount lustre-MDT0000 complete [ 3421.761415] LustreError: 114727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749665 with bad export cookie 6941294852052564978 [ 3421.782791] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3421.785630] LustreError: 114727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3428.473360] Lustre: server umount lustre-MDT0001 complete [ 3447.263278] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.379262] Lustre: server umount lustre-OST0000 complete [ 3457.875816] Lustre: server umount lustre-OST0001 complete [ 3465.857145] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 3476.230418] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3522.238335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.707768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3522.759776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3522.900425] Lustre: lustre-MDT0000: new disk, initializing [ 3523.008487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.020891] Lustre: Skipped 3 previous similar messages [ 3523.061053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3529.131485] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3537.625792] Lustre: server umount lustre-MDT0000 complete [ 3548.431436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.753300] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3548.770690] Lustre: 122054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3548.811715] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3554.566955] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3560.676600] Lustre: server umount lustre-MDT0000 complete [ 3570.176208] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:23:32 (1786749812) [ 3572.182264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3574.337876] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:23:36 (1786749816) [ 3592.862852] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3602.775264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.706891] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3619.403809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.520964] Lustre: 124102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3619.560836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3619.563304] Lustre: Skipped 1 previous similar message [ 3619.619520] Lustre: lustre-MDT0001: new disk, initializing [ 3619.729836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3619.735584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3626.294500] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3632.562544] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3642.963158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3643.342899] Lustre: lustre-OST0000: new disk, initializing [ 3643.347327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3643.353947] Lustre: 125683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3644.506515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3644.520845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3644.725976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3651.751426] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3664.816976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3665.091061] Lustre: lustre-OST0001: new disk, initializing [ 3665.095340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3665.102488] Lustre: 126553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3666.618140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3666.626598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3666.799406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3672.460953] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3682.591861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3686.478815] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4095.885720] Lustre: 132969:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4095.893458] Lustre: 132969:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4095.897872] Lustre: 132969:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4095.907529] Lustre: 132969:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4095.913930] Lustre: 132969:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4095.923850] Lustre: 132969:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4377.131749] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:36:58 (1786750618) [ 4393.442368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4393.460872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.478217] Lustre: Skipped 5 previous similar messages [ 4393.485448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4393.492426] Lustre: Skipped 8 previous similar messages [ 4399.660674] Lustre: server umount lustre-MDT0000 complete [ 4402.150619] LustreError: 127359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.161126] LustreError: 127359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4404.138577] LustreError: 124842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750647 with bad export cookie 6941294852052632668 [ 4404.140647] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.153548] LustreError: 124842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4407.266984] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.267969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4407.287265] Lustre: Skipped 4 previous similar messages [ 4407.289921] Lustre: Skipped 4 previous similar messages [ 4411.065459] Lustre: server umount lustre-MDT0001 complete [ 4422.302664] Lustre: server umount lustre-OST0000 complete [ 4432.681455] Lustre: server umount lustre-OST0001 complete [ 4440.587589] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 4451.230614] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4491.089735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.535582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4491.610721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4491.727736] Lustre: lustre-MDT0000: new disk, initializing [ 4491.861089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.868031] Lustre: Skipped 5 previous similar messages [ 4491.898127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4497.596128] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 4510.253385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.419961] Lustre: 136023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4510.493319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4510.497868] Lustre: Skipped 1 previous similar message [ 4510.643904] Lustre: lustre-MDT0001: new disk, initializing [ 4510.859692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4510.881180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4517.641629] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 4529.513282] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4535.470176] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4542.716521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4543.014263] Lustre: lustre-OST0000: new disk, initializing [ 4543.018151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4543.024199] Lustre: 137993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4543.135424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4543.140524] Lustre: Skipped 1 previous similar message [ 4545.085019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4545.105606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4545.274518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4550.615959] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 4561.969966] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4573.441836] Lustre: lustre-MDD0000: changelog on [ 4576.331489] Lustre: lustre-MDD0001: changelog on [ 6535.751911] Lustre: lustre-MDD0001: changelog off [ 6537.318041] Lustre: 136031:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6538.138552] Lustre: lustre-MDD0000: changelog off [ 6545.381704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6545.413101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6545.422051] Lustre: Skipped 4 previous similar messages [ 6549.454093] Lustre: server umount lustre-OST0000 complete [ 6555.618859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.639450] Lustre: Skipped 1 previous similar message [ 6555.650320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.666264] Lustre: Skipped 1 previous similar message [ 6560.737170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6560.747581] Lustre: Skipped 1 previous similar message [ 6561.289741] Lustre: server umount lustre-MDT0000 complete [ 6565.869135] LustreError: 136034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6565.883843] LustreError: 136034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6566.289917] LustreError: 136014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752810 with bad export cookie 6941294852053409885 [ 6566.316184] LustreError: 136014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6566.320295] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6566.944757] Lustre: server umount lustre-MDT0001 complete [ 6578.327953] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 6581.585953] Key type lgssc unregistered [ 6581.998627] LNet: 147152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6582.007043] LNetError: 147152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6583.078917] LNet: Removed LNI 192.168.201.132@tcp [ 6583.941220] Key type .llcrypt unregistered [ 6583.946083] Key type ._llcrypt unregistered [ 6605.564199] Key type ._llcrypt registered [ 6605.571476] Key type .llcrypt registered [ 6605.712639] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 6622.171800] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 6623.513102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6623.578922] alg: No test for adler32 (adler32-zlib) [ 6625.016878] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 6625.332488] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 6627.063272] Key type lgssc registered [ 6628.491300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6671.474906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6671.523387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.818978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6672.845468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6672.921849] Lustre: lustre-MDT0000: new disk, initializing [ 6673.033840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6673.076788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6678.411424] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6690.899686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.111319] Lustre: 150366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6691.219909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6691.222170] Lustre: Skipped 1 previous similar message [ 6691.337897] Lustre: lustre-MDT0001: new disk, initializing [ 6691.492989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6691.608764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6691.621414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6698.599992] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6709.473465] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6714.845565] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6722.821885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.442766] Lustre: lustre-OST0000: new disk, initializing [ 6723.451104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6723.460655] Lustre: 152336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6723.681341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.858940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6730.874711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6730.991731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6731.045667] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6743.714983] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6751.592365] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_module obdecho/obdecho [ 6752.930768] LustreError: 153340:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6752.935387] LustreError: 153340:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6754.916493] LustreError: 153436:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6772.202417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6772.220454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6772.231260] Lustre: Skipped 1 previous similar message [ 6772.247811] Lustre: Skipped 2 previous similar messages [ 6774.923419] Lustre: server umount lustre-MDT0000 complete [ 6779.228480] LustreError: 151160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753023 with bad export cookie 14083410124309968799 [ 6779.229149] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.239236] LustreError: 151160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6779.631737] Lustre: server umount lustre-MDT0001 complete [ 6794.478251] Lustre: server umount lustre-OST0000 complete [ 6817.721996] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 6826.970762] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 6865.551500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.805540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6865.836182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6865.960287] Lustre: lustre-MDT0000: new disk, initializing [ 6866.040352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.060549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6871.415532] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6883.886962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.001161] Lustre: 157156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6884.049283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6884.051857] Lustre: Skipped 1 previous similar message [ 6884.121477] Lustre: lustre-MDT0001: new disk, initializing [ 6884.248074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6884.333633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6884.359271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6890.161892] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6900.339590] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6906.283658] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6912.952256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6913.320542] Lustre: lustre-OST0000: new disk, initializing [ 6913.325588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6913.338806] Lustre: 159122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6913.482807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6919.848360] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6920.286189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6920.303562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6920.410659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6929.516451] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6933.763798] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6934.073716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6938.562831] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6938.724558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6941.158340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.159059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6941.167272] Lustre: Skipped 1 previous similar message [ 6941.171493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6941.184578] LustreError: Skipped 1 previous similar message [ 6941.194149] Lustre: Skipped 1 previous similar message [ 6946.282213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6946.287253] Lustre: Skipped 1 previous similar message [ 6947.127263] Lustre: server umount lustre-OST0000 complete [ 6950.882922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6950.888112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.900170] Lustre: Skipped 1 previous similar message [ 6950.905740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.512647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.520522] Lustre: Skipped 3 previous similar messages [ 6956.766166] Lustre: server umount lustre-MDT0000 complete [ 6960.629718] LustreError: 157148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753204 with bad export cookie 14083410124309969758 [ 6960.637702] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.642265] LustreError: 157148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6961.011736] Lustre: server umount lustre-MDT0001 complete [ 6964.453558] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:20:06 (1786753206) [ 6966.044305] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6967.703427] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:20:09 (1786753209) [ 6974.522835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6975.025615] LustreError: 161534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.056546] LustreError: 161534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6975.162136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6979.485435] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6980.580928] LustreError: 161535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.676645] LustreError: 161534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.509441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.650139] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6999.772900] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7005.514604] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7012.361750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7012.674027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.680148] Lustre: Skipped 1 previous similar message [ 7019.494708] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7027.443843] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.434606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.451536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7038.456558] Lustre: Skipped 2 previous similar messages [ 7038.468055] Lustre: Skipped 2 previous similar messages [ 7039.316714] Lustre: server umount lustre-MDT0000 complete [ 7043.287540] LustreError: 161515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753287 with bad export cookie 14083410124309970640 [ 7043.297901] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7043.304681] LustreError: 161515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7043.552902] LustreError: 161529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.556644] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.590516] Lustre: Skipped 1 previous similar message [ 7048.673410] LustreError: 161531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.681451] LustreError: 161531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7049.872438] Lustre: server umount lustre-MDT0001 complete [ 7055.656742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7068.411635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.839417] LustreError: 165377:0:(llog.c:1655:llog_backup()) MGC192.168.201.132@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7068.851526] Lustre: 165377:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.132@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7073.258229] LustreError: 165382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.279680] LustreError: 165382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7093.728669] LustreError: 165382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.756977] LustreError: 165382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 7129.058629] LustreError: 165384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.079986] LustreError: 165384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7129.568086] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7129.580420] Lustre: 165377:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.132@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7170.558947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7175.181774] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7175.675174] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7183.962770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.330813] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7194.610916] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7200.149676] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7201.671388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7207.285553] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7208.615296] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7230.441794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.442341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7230.458546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7230.484833] Lustre: Skipped 7 previous similar messages [ 7231.259462] Lustre: server umount lustre-MDT0000 complete [ 7235.222576] LustreError: 164316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753479 with bad export cookie 14083410124309977094 [ 7235.236154] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7235.237773] LustreError: 164316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7235.548402] Lustre: server umount lustre-MDT0001 complete [ 7249.956482] Lustre: server umount lustre-OST0000 complete [ 7272.131456] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 7282.592882] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 7321.622174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.999442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7322.041859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7322.133450] Lustre: lustre-MDT0000: new disk, initializing [ 7322.269252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7322.272107] Lustre: Skipped 1 previous similar message [ 7322.297748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7327.318803] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7338.391799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7338.592909] Lustre: 170481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7339.081813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7344.339783] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7354.316642] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7361.565122] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7369.790266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7370.335665] Lustre: lustre-OST0000: new disk, initializing [ 7370.341627] Lustre: Skipped 1 previous similar message [ 7370.351865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7370.370282] Lustre: Skipped 2 previous similar messages [ 7370.376775] Lustre: 172451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7377.495885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7377.509190] Lustre: Skipped 1 previous similar message [ 7377.517938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7377.631396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7377.812240] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7388.887392] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7394.393695] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7394.605322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7399.553436] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7399.819113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7403.489083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7403.503355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.511238] LustreError: Skipped 1 previous similar message [ 7403.513433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7403.529905] Lustre: Skipped 3 previous similar messages [ 7403.550102] Lustre: Skipped 1 previous similar message [ 7416.287723] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7416.571747] Lustre: server umount lustre-OST0000 complete [ 7420.898883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7435.231214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7435.431730] Lustre: server umount lustre-MDT0000 complete [ 7439.002594] LustreError: 170473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753682 with bad export cookie 14083410124309993929 [ 7439.011100] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7439.021462] LustreError: 170473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7439.337161] Lustre: server umount lustre-MDT0001 complete [ 7449.914177] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:28:11 (1786753691) [ 7458.474425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.117223] LustreError: 174864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7459.137651] LustreError: 174864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 7463.934799] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7474.545358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7479.656676] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7489.686751] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7496.897617] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7503.995701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7511.503489] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7522.256800] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7545.564144] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:29:46 (1786753786) [ 7554.532431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7554.537434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7554.554370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7554.554383] Lustre: Skipped 11 previous similar messages [ 7554.577753] Lustre: Skipped 3 previous similar messages [ 7558.119862] Lustre: server umount lustre-OST0000 complete [ 7569.221291] Lustre: server umount lustre-MDT0000 complete [ 7573.067975] LustreError: 174845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753816 with bad export cookie 14083410124309994811 [ 7573.070758] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7573.077132] LustreError: 174845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7573.476068] Lustre: server umount lustre-MDT0001 complete [ 7585.379401] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 7588.394795] Key type lgssc unregistered [ 7588.771288] LNet: 178624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7588.796744] LNetError: 178624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7588.813309] LNet: Removed LNI 192.168.201.132@tcp [ 7589.763584] Key type .llcrypt unregistered [ 7589.766260] Key type ._llcrypt unregistered [ 7610.997592] Key type ._llcrypt registered [ 7610.999636] Key type .llcrypt registered [ 7611.083911] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 7612.112227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7612.166096] alg: No test for adler32 (adler32-zlib) [ 7613.256359] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 7613.486099] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 7615.224047] Key type lgssc registered [ 7616.594275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7625.025276] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7625.053089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.799892] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7645.919751] LustreError: 179868:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.132@tcp: failed processing log, type 4: rc = -110 [ 7651.039495] LustreError: 179868:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7651.072477] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7651.088975] LustreError: MGC192.168.201.132@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7651.120040] LustreError: 179868:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7651.139736] LustreError: 179868:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7651.147527] LustreError: 179868:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7651.260215] Lustre: server umount lustre-OST0000 complete [ 7651.262347] LustreError: 179868:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7660.663762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.082336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7667.304809] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7671.278160] LustreError: 180368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.303450] LustreError: 180368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7676.385592] LustreError: 180369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7677.606635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.138367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7683.421046] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7691.754284] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7697.869485] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7705.118177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7705.367839] LustreError: 182186:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7705.379118] Lustre: 182186:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.132@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7705.612894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.629891] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7723.348528] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7728.101472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.119924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7728.131348] Lustre: Skipped 1 previous similar message [ 7733.217259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7733.230465] Lustre: Skipped 2 previous similar messages [ 7738.339781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7738.346961] Lustre: Skipped 2 previous similar messages [ 7740.383140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7740.655617] Lustre: server umount lustre-MDT0000 complete [ 7743.458329] LustreError: 180365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.493285] LustreError: 180365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7744.711892] LustreError: 180347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753988 with bad export cookie 1550715981209122887 [ 7744.732292] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7745.089635] Lustre: server umount lustre-MDT0001 complete [ 7759.729406] Lustre: server umount lustre-OST0000 complete [ 7771.020618] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 7782.932183] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 7823.939676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.425469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7824.470610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7824.621294] Lustre: lustre-MDT0000: new disk, initializing [ 7824.746948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7824.783921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7829.845513] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7841.577905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.728747] Lustre: 185713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7841.791784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7841.795313] Lustre: Skipped 1 previous similar message [ 7841.852676] Lustre: lustre-MDT0001: new disk, initializing [ 7842.052454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7842.134028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7842.157221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7846.943308] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7857.598484] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7864.551342] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7871.841216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7872.179193] Lustre: lustre-OST0000: new disk, initializing [ 7872.185132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7872.192348] Lustre: 187685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7872.271609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7879.227142] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7880.326909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7880.344329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7880.436806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7889.446193] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7894.794232] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7895.124785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7900.147068] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7900.345726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7906.274722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7906.284068] Lustre: Skipped 2 previous similar messages [ 7906.285883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7906.292991] LustreError: Skipped 1 previous similar message [ 7906.295394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7906.307680] Lustre: Skipped 1 previous similar message [ 7908.796273] Lustre: server umount lustre-OST0000 complete [ 7913.953182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7913.966709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.983340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.518557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.752520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7926.760751] Lustre: Skipped 4 previous similar messages [ 7927.263791] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7927.722967] Lustre: server umount lustre-MDT0000 complete [ 7931.878708] LustreError: 185724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.902246] LustreError: 185724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7932.508666] LustreError: 185705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754176 with bad export cookie 1550715981209123608 [ 7932.509616] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7932.523455] LustreError: 185705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7933.198653] Lustre: server umount lustre-MDT0001 complete [ 7944.651889] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:36:26 (1786754186) [ 7965.596785] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 7969.423926] Key type lgssc unregistered [ 7969.798767] LNet: 190636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7969.810891] LNetError: 190636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7969.838868] LNet: Removed LNI 192.168.201.132@tcp [ 7971.158815] Key type .llcrypt unregistered [ 7971.161660] Key type ._llcrypt unregistered [ 7990.197510] Key type ._llcrypt registered [ 7990.199071] Key type .llcrypt registered [ 7990.295375] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 7991.439716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7991.513693] alg: No test for adler32 (adler32-zlib) [ 7992.616570] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 7992.938539] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 7994.671197] Key type lgssc registered [ 7995.737157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8003.893920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8003.940982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8005.857138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8011.698397] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8015.352987] LustreError: 191855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8015.378300] LustreError: 191855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8020.465978] LustreError: 191856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8021.357345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.704993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8026.659101] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8036.399807] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8044.452269] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8053.852790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8054.156789] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8054.158912] LustreError: 193719:0:(llog.c:1696:llog_backup()) MGC192.168.201.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8054.179068] Lustre: 193719:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.132@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8054.355375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8062.344752] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8070.263719] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8077.281541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.286674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8077.307144] Lustre: Skipped 2 previous similar messages [ 8077.325984] Lustre: Skipped 2 previous similar messages [ 8079.522341] Lustre: server umount lustre-MDT0000 complete [ 8082.403719] LustreError: 191852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.432796] LustreError: 191852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8082.982153] LustreError: 191835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754326 with bad export cookie 987145261851968190 [ 8082.989929] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8082.993870] LustreError: 191835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8083.441750] Lustre: server umount lustre-MDT0001 complete [ 8097.889944] Lustre: server umount lustre-OST0000 complete [ 8107.848277] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 8119.688774] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8160.272853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.608277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8160.647420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8160.716891] Lustre: lustre-MDT0000: new disk, initializing [ 8160.795931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.824446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8165.028714] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8176.374736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.537628] Lustre: 197246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8176.614857] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8176.617531] Lustre: Skipped 1 previous similar message [ 8176.620412] LustreError: 198029:0:(llog.c:1696:llog_backup()) MGC192.168.201.132@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8176.626592] LustreError: 198029:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8176.631135] Lustre: 198029:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.132@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8176.652728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8176.661665] Lustre: Skipped 1 previous similar message [ 8176.763469] Lustre: lustre-MDT0001: new disk, initializing [ 8176.905439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8176.947400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8176.955625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8181.117836] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8190.129956] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8196.244555] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8202.792396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8203.039869] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8203.042459] Lustre: Skipped 1 previous similar message [ 8203.045601] LustreError: 199193:0:(llog.c:1696:llog_backup()) MGC192.168.201.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8203.052247] LustreError: 199193:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8203.056511] Lustre: 199193:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.132@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8203.062576] Lustre: 199193:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8203.080631] Lustre: lustre-OST0000: new disk, initializing [ 8203.084276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8203.089163] Lustre: 199218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8203.182558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.511445] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8211.076764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8211.083686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8211.203616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8219.864792] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8225.979168] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8226.284687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8231.515671] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8231.718619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8237.024626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8237.037763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.060645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.061917] Lustre: Skipped 1 previous similar message [ 8237.081302] Lustre: Skipped 1 previous similar message [ 8239.654086] Lustre: server umount lustre-OST0000 complete [ 8247.268191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.269600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.277230] Lustre: Skipped 1 previous similar message [ 8247.290155] Lustre: Skipped 1 previous similar message [ 8250.245830] Lustre: server umount lustre-MDT0000 complete [ 8252.395782] LustreError: 197258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.407423] LustreError: 197258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8254.077242] LustreError: 198042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754497 with bad export cookie 987145261851968904 [ 8254.083333] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.092943] LustreError: 198042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8254.375201] Lustre: server umount lustre-MDT0001 complete [ 8265.499538] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:41:47 (1786754507) [ 8283.713308] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8295.232449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8295.815250] LustreError: 202346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8295.854853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.725890] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8306.147214] LustreError: 202347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8306.185925] LustreError: 202347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8311.231697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8316.449351] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8319.537435] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8326.892097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8333.586746] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8343.128625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8343.388695] Lustre: lustre-OST0001: new disk, initializing [ 8343.394971] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8343.413582] Lustre: 204651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8343.569180] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8343.576400] Lustre: Skipped 2 previous similar messages [ 8349.716301] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8352.371466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8352.384481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8352.488772] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8361.783710] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8369.794754] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8372.118764] Lustre: DEBUG MARKER: ADD OST3 [ 8377.312182] loop: module loaded [ 8377.411473] loop0: detected capacity change from 0 to 204800000 [ 8377.810099] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8377.828501] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8377.842190] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8385.426244] loop0: detected capacity change from 0 to 204800000 [ 8385.904754] Lustre: DEBUG MARKER: STOP OST3 [ 8392.364471] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8392.680791] Lustre: lustre-OST0003: new disk, initializing [ 8392.693796] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8392.709000] Lustre: 206673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8394.194555] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8394.401584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8394.425334] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8394.483727] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8396.258058] LustreError: 206788:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8396.910989] Lustre: Failing over lustre-OST0003 [ 8397.102932] Lustre: server umount lustre-OST0003 complete [ 8399.340986] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8399.360213] LustreError: Skipped 1 previous similar message [ 8399.365624] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.422219] LustreError: 203830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8399.458931] LustreError: 203830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8401.163966] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8410.236286] Lustre: DEBUG MARKER: START OST3 again [ 8418.897287] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.405197] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8419.422699] Lustre: Skipped 1 previous similar message [ 8419.449265] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8420.050820] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8421.271950] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8421.274297] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8428.040059] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8453.506654] Lustre: Failing over lustre-OST0003 [ 8453.752509] Lustre: server umount lustre-OST0003 complete [ 8455.168705] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.185618] Lustre: Skipped 1 previous similar message [ 8455.195758] LustreError: 203830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.218512] LustreError: 203830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8475.616165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8475.631763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.631784] Lustre: Skipped 1 previous similar message [ 8475.638750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8480.305277] Lustre: server umount lustre-MDT0000 complete [ 8483.908267] LustreError: 205420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754727 with bad export cookie 987145261851969779 [ 8483.931026] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8483.935749] LustreError: 205420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8485.868441] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8485.875010] Lustre: Skipped 4 previous similar messages [ 8490.300111] Lustre: server umount lustre-MDT0001 complete [ 8494.341755] Lustre: server umount lustre-OST0000 complete [ 8498.607551] Lustre: server umount lustre-OST0001 complete [ 8504.942737] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 8513.533349] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8556.724519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8557.054991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8557.113553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8557.291131] Lustre: lustre-MDT0000: new disk, initializing [ 8557.451064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.489373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8563.491901] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8574.766528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.834414] Lustre: 212123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8574.884560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8574.887112] Lustre: Skipped 1 previous similar message [ 8574.960474] Lustre: lustre-MDT0001: new disk, initializing [ 8575.167481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8575.184977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8579.290878] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8589.058693] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8594.764194] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8601.235170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8601.470963] Lustre: 214093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8603.639235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8603.790108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8608.079839] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8618.929854] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8624.635478] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8624.856475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8631.019306] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8631.462624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8634.337323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.345390] Lustre: Skipped 6 previous similar messages [ 8634.346768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.349889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8634.357869] Lustre: Skipped 3 previous similar messages [ 8639.478630] LustreError: 214076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8639.502691] LustreError: 214076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8639.598594] Lustre: server umount lustre-OST0000 complete [ 8650.109146] Lustre: server umount lustre-MDT0000 complete [ 8654.789649] LustreError: 212116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754898 with bad export cookie 987145261851972075 [ 8654.812431] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8654.814078] LustreError: 212116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8655.176152] Lustre: server umount lustre-MDT0001 complete [ 8659.665308] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:48:21 (1786754901) [ 8678.848860] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 8690.429736] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8734.963844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.580304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8735.677501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8735.683376] Lustre: Skipped 1 previous similar message [ 8735.906219] Lustre: lustre-MDT0000: new disk, initializing [ 8735.916668] Lustre: Skipped 1 previous similar message [ 8736.100983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8736.108238] Lustre: Skipped 1 previous similar message [ 8740.282343] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8753.205136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8753.369891] Lustre: 218705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8753.701369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.378026] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8768.479729] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8773.824990] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8781.106962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8781.404981] Lustre: 220678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8782.593280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8788.437872] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8797.995764] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8802.360924] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8802.560195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8807.674742] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8807.919908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8813.536182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8813.548275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.550298] LustreError: Skipped 2 previous similar messages [ 8813.557267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8813.557273] Lustre: Skipped 2 previous similar messages [ 8813.584158] Lustre: Skipped 3 previous similar messages [ 8816.725285] Lustre: server umount lustre-OST0000 complete [ 8826.884921] Lustre: server umount lustre-MDT0000 complete [ 8828.897386] LustreError: 221000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.924510] LustreError: 221000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8830.480082] LustreError: 218696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755074 with bad export cookie 987145261851972957 [ 8830.486116] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8830.494666] LustreError: 218696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8830.838509] Lustre: server umount lustre-MDT0001 complete [ 8840.005122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8840.713991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8840.725724] Lustre: Skipped 5 previous similar messages [ 8845.655893] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8854.293382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8859.515157] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8866.456544] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8872.601969] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8878.434529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8886.109931] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8893.880618] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8905.696040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8905.710345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.729881] Lustre: Skipped 2 previous similar messages [ 8905.742906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8905.755876] Lustre: Skipped 3 previous similar messages [ 8909.934587] Lustre: server umount lustre-OST0000 complete [ 8919.547420] Lustre: server umount lustre-MDT0000 complete [ 8923.087305] LustreError: 222874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755166 with bad export cookie 987145261851973839 [ 8923.103132] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8923.105685] LustreError: 222874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8923.364926] Lustre: server umount lustre-MDT0001 complete [ 9032.704165] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 9035.305893] Key type lgssc unregistered [ 9035.524216] LNet: 226558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9035.539091] LNetError: 226558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9035.547854] LNet: Removed LNI 192.168.201.132@tcp [ 9036.442172] Key type .llcrypt unregistered [ 9036.445737] Key type ._llcrypt unregistered [ 9063.631333] Key type ._llcrypt registered [ 9063.633361] Key type .llcrypt registered [ 9063.761281] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 9079.920305] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9081.150915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9081.265490] alg: No test for adler32 (adler32-zlib) [ 9082.321340] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 9082.517797] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 9084.159247] Key type lgssc registered [ 9085.342780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9123.400393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9123.435915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.746878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9124.787330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9124.895779] Lustre: lustre-MDT0000: new disk, initializing [ 9125.066122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9125.085608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9129.390622] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9139.506627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.655559] Lustre: 229920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9139.733865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9139.740761] Lustre: Skipped 1 previous similar message [ 9139.885083] Lustre: lustre-MDT0001: new disk, initializing [ 9139.974716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9140.026865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9140.052252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9145.845248] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9155.714183] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9161.878383] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9168.734468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9169.060809] Lustre: lustre-OST0000: new disk, initializing [ 9169.064047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9169.070722] Lustre: 231889:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9169.151228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9175.504466] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9175.622819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9175.635818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9175.756332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9185.216587] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9190.127362] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9190.300363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9194.845902] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9195.088134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9201.632473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9201.640304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.644072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9201.651467] Lustre: Skipped 1 previous similar message [ 9201.662308] Lustre: Skipped 1 previous similar message [ 9203.516832] Lustre: server umount lustre-OST0000 complete [ 9211.873267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9211.875609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9211.890888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9211.908636] Lustre: Skipped 1 previous similar message [ 9211.935207] Lustre: Skipped 1 previous similar message [ 9216.993241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9216.997150] Lustre: Skipped 1 previous similar message [ 9222.111112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9222.112456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9222.127532] Lustre: Skipped 1 previous similar message [ 9222.298614] Lustre: server umount lustre-MDT0000 complete [ 9226.480898] LustreError: 229913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755470 with bad export cookie 8080274567948798139 [ 9226.491843] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9226.503623] LustreError: 229913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9226.910058] Lustre: server umount lustre-MDT0001 complete [ 9254.305433] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9265.666704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9266.145127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9271.033345] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9271.277296] LustreError: 234837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9271.315488] LustreError: 234837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9275.368344] LustreError: 234836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9280.481213] LustreError: 234837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9281.464956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.013071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9287.119417] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9291.062983] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9299.089437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9299.367872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9307.747784] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9317.570666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9317.851326] Lustre: lustre-OST0001: new disk, initializing [ 9317.857666] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9317.873754] Lustre: 237144:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9317.997808] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9325.036279] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9327.779995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9327.789812] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9327.899535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9337.007613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9344.608213] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9357.522856] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:59:59 (1786755599) [ 9363.936628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9363.940184] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.942068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9363.960981] Lustre: Skipped 3 previous similar messages [ 9374.175171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9374.181431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9374.187438] Lustre: Skipped 9 previous similar messages [ 9374.356891] Lustre: server umount lustre-MDT0000 complete [ 9377.993709] LustreError: 237911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755621 with bad export cookie 8080274567948799021 [ 9377.997511] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9378.004552] LustreError: 237911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9378.260860] Lustre: server umount lustre-MDT0001 complete [ 9391.866572] Lustre: server umount lustre-OST0000 complete [ 9395.679842] Lustre: 228146:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786755623/real 1786755623] req@ffff97ca3bd6f100 x1873548752832128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786755639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9395.728920] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.027370] Lustre: server umount lustre-OST0001 complete [ 9403.861805] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 9415.879715] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9455.621692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.938452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9455.966413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9456.071175] Lustre: lustre-MDT0000: new disk, initializing [ 9456.211411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.234787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9461.914812] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9474.694143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.025224] Lustre: 241105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9475.135397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9475.142604] Lustre: Skipped 1 previous similar message [ 9475.315509] Lustre: lustre-MDT0001: new disk, initializing [ 9475.462831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9475.476256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9481.837293] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9493.091087] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9499.710066] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9507.907883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9508.321610] Lustre: lustre-OST0000: new disk, initializing [ 9508.326881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9508.336614] Lustre: 243076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9509.607047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9509.621068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9509.800303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9515.040830] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9526.173642] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9531.626618] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9531.899879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9538.295596] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9538.558587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9545.703307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9545.704558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9545.719897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9545.723267] Lustre: Skipped 1 previous similar message [ 9545.736720] Lustre: Skipped 1 previous similar message [ 9546.961406] Lustre: server umount lustre-OST0000 complete [ 9552.361032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9552.366514] LustreError: Skipped 1 previous similar message [ 9555.938354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9555.955425] Lustre: Skipped 1 previous similar message [ 9558.098727] Lustre: server umount lustre-MDT0000 complete [ 9561.068022] LustreError: 241116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9561.087429] LustreError: 241116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9561.696428] LustreError: 241096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755805 with bad export cookie 8080274567948801156 [ 9561.699231] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9561.716016] LustreError: 241096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9562.069185] Lustre: server umount lustre-MDT0001 complete [ 9570.532311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.163270] LustreError: 245294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9571.229179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9571.241612] Lustre: Skipped 2 previous similar messages [ 9575.794384] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9580.519085] LustreError: 245295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.539824] LustreError: 245295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9584.361455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.020557] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9598.403818] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9605.867055] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9613.503352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9620.786739] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9628.009834] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9773.026917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9773.032656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9773.066278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9773.088967] Lustre: Skipped 4 previous similar messages [ 9786.335177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9786.727658] Lustre: server umount lustre-OST0000 complete [ 9797.548826] Lustre: server umount lustre-MDT0000 complete [ 9798.627990] LustreError: 245294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.651815] LustreError: 245294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9801.618859] LustreError: 245275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756045 with bad export cookie 8080274567948802038 [ 9801.633385] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.944270] Lustre: server umount lustre-MDT0001 complete [ 9897.511910] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 9901.297420] Key type lgssc unregistered [ 9901.690913] LNet: 248908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9901.702195] LNetError: 248908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9901.746766] LNet: Removed LNI 192.168.201.132@tcp [ 9902.945135] Key type .llcrypt unregistered [ 9902.948124] Key type ._llcrypt unregistered [ 9931.456888] Key type ._llcrypt registered [ 9931.458702] Key type .llcrypt registered [ 9931.536177] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 9949.302905] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9950.983951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9950.998177] alg: No test for adler32 (adler32-zlib) [ 9952.208560] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 9952.526131] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 9954.239177] Key type lgssc registered [ 9955.629160] Lustre: Echo OBD driver; http://www.lustre.org/ [10005.982665] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10016.278291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10016.307115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.657875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10017.700306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10017.836143] Lustre: lustre-MDT0000: new disk, initializing [10017.956328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10017.976705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10021.869745] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10032.505711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10032.634542] Lustre: 252979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10032.675722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10032.679870] Lustre: Skipped 1 previous similar message [10032.760865] Lustre: lustre-MDT0001: new disk, initializing [10032.855164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10032.910703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10032.917832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10037.075756] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10041.724402] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10047.985198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.219223] Lustre: lustre-OST0000: new disk, initializing [10048.222741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10048.228136] Lustre: 254609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10048.335216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10049.804438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10049.820918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10049.943715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10054.188650] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10064.376985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10064.619896] Lustre: lustre-OST0001: new disk, initializing [10064.621954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10064.626345] Lustre: 255482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10064.717392] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10070.414360] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10074.216242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10074.233042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10074.393487] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10081.558426] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10089.511374] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10102.863710] Lustre: server umount lustre-MDT0000 complete [10104.800203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10104.818302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.341407] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.688940] LustreError: 255481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756350 with bad export cookie 17237644974289102184 [10106.706910] LustreError: 255481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10106.712072] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10107.012989] Lustre: server umount lustre-MDT0001 complete [10122.113204] Lustre: server umount lustre-OST0000 complete [10136.983676] Lustre: server umount lustre-OST0001 complete [10151.651175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10161.702931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10170.518944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10180.094606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10200.862473] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10211.146906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10211.170919] Lustre: lustre-MDT0000: reset Object Index mappings [10211.698859] LustreError: 260492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.707369] LustreError: 260492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10211.792345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10216.395518] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10216.931667] LustreError: 260493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.048424] LustreError: 260492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10225.019958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10225.038818] Lustre: lustre-MDT0001: reset Object Index mappings [10225.911163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10231.223425] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10234.989429] Lustre: 261634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10243.081890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10243.418463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10250.492891] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10250.540587] LustreError: 261992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.862181] LustreError: 261990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.915604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10260.277603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10266.141984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10268.583924] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10276.245206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10280.107499] Lustre: 263512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10286.562651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.573611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10286.587324] Lustre: Skipped 5 previous similar messages [10286.603710] Lustre: Skipped 3 previous similar messages [10287.679646] Lustre: server umount lustre-MDT0000 complete [10291.558284] LustreError: 260474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756535 with bad export cookie 17237644974289108974 [10291.562408] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10291.565373] LustreError: 260474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10291.683505] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10291.688248] LustreError: 260489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10291.707979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10291.717133] LustreError: 260489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10291.727438] Lustre: Skipped 1 previous similar message [10291.950945] Lustre: server umount lustre-MDT0001 complete [10297.384626] Lustre: server umount lustre-OST0000 complete [10301.568660] Lustre: server umount lustre-OST0001 complete [10323.180801] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:16:04 (1786756564) [10339.643152] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [10351.507368] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10390.415968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10390.665319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10390.682627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10390.754911] Lustre: lustre-MDT0000: new disk, initializing [10390.869655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10390.873701] Lustre: Skipped 1 previous similar message [10390.896951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10396.594770] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10408.527937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.693391] Lustre: 267765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10408.745873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10408.753947] Lustre: Skipped 1 previous similar message [10408.855481] Lustre: lustre-MDT0001: new disk, initializing [10409.013235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10409.027604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10414.852542] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10426.663430] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10435.524319] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10445.061543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10445.382501] Lustre: lustre-OST0000: new disk, initializing [10445.385928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10445.391660] Lustre: 269734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10447.075697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10447.090642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10447.204496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10452.474170] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10462.378881] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10467.656497] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10468.013965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10472.490663] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10472.747581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10478.064705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.071345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10478.084668] Lustre: Skipped 1 previous similar message [10478.093129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10478.115218] LustreError: Skipped 1 previous similar message [10478.141030] Lustre: Skipped 1 previous similar message [10483.172267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10483.177336] Lustre: Skipped 1 previous similar message [10488.290141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10488.300180] Lustre: Skipped 1 previous similar message [10489.311510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10489.673565] Lustre: server umount lustre-OST0000 complete [10495.968895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10495.983824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.004621] Lustre: Skipped 1 previous similar message [10498.534315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10498.541936] Lustre: Skipped 2 previous similar messages [10500.527733] Lustre: server umount lustre-MDT0000 complete [10503.648209] LustreError: 267777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10503.680539] LustreError: 267777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10505.326406] LustreError: 267757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756749 with bad export cookie 17237644974289111067 [10505.330501] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10505.348324] LustreError: 267757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10505.981455] Lustre: server umount lustre-MDT0001 complete [10532.042777] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10547.160724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.387605] LustreError: 272692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.420794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10547.423727] Lustre: Skipped 2 previous similar messages [10552.293321] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10562.415319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.360133] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10573.319410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10582.642317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10589.685609] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10602.226051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10602.647588] Lustre: lustre-OST0001: new disk, initializing [10602.656462] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10602.673483] Lustre: 275005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10604.140018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10604.143185] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10604.289352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10610.109603] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10622.379682] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10626.575232] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10631.732595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10631.861496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10639.328412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10639.346085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10639.367877] Lustre: Skipped 1 previous similar message [10639.378614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10644.975578] Lustre: server umount lustre-MDT0000 complete [10648.081422] LustreError: 272674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756891 with bad export cookie 17237644974289111949 [10648.087189] LustreError: 272674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10648.092493] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10648.301682] Lustre: server umount lustre-MDT0001 complete [10661.614097] Lustre: server umount lustre-OST0000 complete [10664.419733] Lustre: 250486:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756892/real 1786756892] req@ffff97c907cb2300 x1873549665216768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786756908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10666.158163] Lustre: server umount lustre-OST0001 complete [10691.965689] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10702.128044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10702.393434] LustreError: 278062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10702.409439] LustreError: 278062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10702.467102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10702.471150] Lustre: Skipped 3 previous similar messages [10708.538396] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10719.949213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.311969] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10728.440458] Lustre: 279201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10734.573047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10741.819862] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10747.364309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10751.021090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10756.645583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10757.490106] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10765.430553] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10769.409220] Lustre: 281076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10775.679499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10775.773574] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10786.518816] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:23:48 (1786757028) [10802.663180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10802.672734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.692411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10802.694022] Lustre: Skipped 5 previous similar messages [10802.707834] Lustre: Skipped 5 previous similar messages [10808.852168] Lustre: server umount lustre-MDT0000 complete [10812.474503] LustreError: 278043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757056 with bad export cookie 17237644974289118501 [10812.479082] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10812.484927] LustreError: 278043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10819.012282] Lustre: server umount lustre-MDT0001 complete [10829.306240] Lustre: server umount lustre-OST0000 complete [10834.331220] Lustre: server umount lustre-OST0001 complete [10841.996579] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [10854.528182] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10908.895740] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10922.211336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.669979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10922.745620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10922.898920] Lustre: lustre-MDT0000: new disk, initializing [10923.142894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10929.496418] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10940.161558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10940.301617] Lustre: 285322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10940.550692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10945.590518] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10959.100084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.437452] Lustre: 286955:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10959.638217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10959.641140] Lustre: Skipped 5 previous similar messages [10961.100025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10967.375764] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10979.465327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10979.655978] Lustre: 287825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10981.217052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10981.227391] Lustre: Skipped 1 previous similar message [10981.440575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10987.112307] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10997.797293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11001.815165] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11001.820501] Lustre: Skipped 1 previous similar message [11017.781396] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:27:39 (1786757259) [11022.040748] Lustre: server umount lustre-MDT0000 complete [11022.309634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.318079] LustreError: 285326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11022.324801] Lustre: Skipped 5 previous similar messages [11022.351232] LustreError: 285326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11022.352636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11025.980070] LustreError: 285313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757269 with bad export cookie 17237644974289126131 [11025.991364] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11025.999775] LustreError: 285313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11026.397868] Lustre: server umount lustre-MDT0001 complete [11039.744924] Lustre: server umount lustre-OST0000 complete [11043.807935] Lustre: 250484:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786757271/real 1786757271] req@ffff97ca3cb19c00 x1873549665369984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786757287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11044.699663] Lustre: server umount lustre-OST0001 complete [11056.444515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11062.873214] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11072.014587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11076.182335] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11085.778451] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11095.488733] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11104.027147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11110.471375] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11118.574110] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11146.633661] loop1: detected capacity change from 0 to 204800000 [11147.059827] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11147.075864] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11147.100996] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11153.049990] loop1: detected capacity change from 0 to 204800000 [11161.195788] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11161.256954] LustreError: 290057:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs7251: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11161.266142] LustreError: 290057:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs7251 [11161.269932] LustreError: 290057:0:(mgs_handler.c:517:mgs_target_reg()) fs7251-MDT0001: Error checking target index: rc = -13 [11161.285357] LustreError: fs7251-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11161.297633] LustreError: 293811:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11161.302843] LustreError: 293811:0:(tgt_mount.c:2129:server_put_super()) no obd fs7251-MDT0001 [11161.309221] LustreError: 293811:0:(tgt_mount.c:116:server_deregister_mount()) fs7251-MDT0001 not registered [11161.381156] Lustre: server umount fs7251-MDT0001 complete [11161.384113] LustreError: 293811:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11171.585836] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.657174] Lustre: Setting parameter fs7251-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs7251-MDT0001 [11171.763885] Lustre: srv-fs7251-MDT0001: No data found on store. Initialize space. [11171.768928] Lustre: Skipped 4 previous similar messages [11171.858396] Lustre: fs7251-MDT0001: new disk, initializing [11171.863107] Lustre: Skipped 3 previous similar messages [11172.052885] Lustre: srv-fs7251-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11172.066161] Lustre: 294277:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7251-MDT0001-mdtlov: can't create new llog: rc = -115 [11172.086789] LustreError: 294277:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7251-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11177.640569] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11184.556166] Lustre: server umount fs7251-MDT0001 complete [11197.929885] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11201.438388] loop1: detected capacity change from 0 to 204800000 [11207.891749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11208.128773] Lustre: 295846:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7251-MDT0001-mdtlov: can't create new llog: rc = -115 [11208.138964] LustreError: 295846:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7251-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11214.811115] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11220.990722] Lustre: server umount fs7251-MDT0001 complete [11233.770645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11233.791166] Lustre: Skipped 4 previous similar messages [11233.801508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11233.815146] Lustre: Skipped 9 previous similar messages [11239.747822] Lustre: server umount lustre-MDT0000 complete [11249.605596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.747147] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11255.147923] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11255.271712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11264.038318] loop1: detected capacity change from 0 to 204800000 [11270.600754] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11271.220089] Lustre: 298295:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7251-MDT0001-mdtlov: can't create new llog: rc = -115 [11271.232284] LustreError: 298295:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7251-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11277.783467] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11280.873801] LustreError: 291901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11280.902796] LustreError: 291901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 106 previous similar messages [11282.916059] Lustre: server umount fs7251-MDT0001 complete [11311.584937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11311.592050] LustreError: Skipped 1 previous similar message [11323.359247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11323.597598] Lustre: server umount lustre-OST0000 complete [11334.609640] Lustre: server umount lustre-MDT0000 complete [11338.810480] LustreError: 290812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757582 with bad export cookie 17237644974289128819 [11338.827492] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11339.282905] Lustre: server umount lustre-MDT0001 complete [11350.137205] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [11352.427052] Key type lgssc unregistered [11352.717434] LNet: 300644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11352.723917] LNetError: 300644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11352.740637] LNet: Removed LNI 192.168.201.132@tcp [11353.629794] Key type .llcrypt unregistered [11353.633103] Key type ._llcrypt unregistered [11372.791447] Key type ._llcrypt registered [11372.793585] Key type .llcrypt registered [11372.909368] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [11374.152606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11374.181919] alg: No test for adler32 (adler32-zlib) [11375.782893] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [11376.072969] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [11377.879200] Key type lgssc registered [11379.894757] Lustre: Echo OBD driver; http://www.lustre.org/ [11391.054311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11391.084530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.077140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11399.512972] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11402.252399] LustreError: 301916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.269944] LustreError: 301916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11407.329896] LustreError: 301917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.659857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11410.113835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11415.334668] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11424.687731] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11431.369041] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11440.326274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.989795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11451.960778] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11454.329264] LustreError: 303747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.557850] LustreError: 303747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11462.116445] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11464.909226] LustreError: 303750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.935413] LustreError: 303750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11475.178538] LustreError: 303749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11475.226393] LustreError: 303749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11486.244404] loop1: detected capacity change from 0 to 204800000 [11486.477751] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11486.489363] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11486.506171] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11492.741513] loop1: detected capacity change from 0 to 204800000 [11493.353537] LustreError: 303750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11493.397777] LustreError: 303750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11500.372330] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11500.538674] Lustre: fs25660-OST0000: new disk, initializing [11500.555631] Lustre: srv-fs25660-OST0000: No data found on store. Initialize space. [11500.583536] Lustre: 305658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11500.765772] Lustre: fs25660-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11509.078995] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11519.698945] Lustre: server umount fs25660-OST0000 complete [11529.192357] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11529.218072] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11532.029783] loop1: detected capacity change from 0 to 204800000 [11534.338569] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11539.445058] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11541.607852] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11542.084372] Lustre: fs25660-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11549.556481] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11560.455292] loop2: detected capacity change from 0 to 204800000 [11560.719446] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11560.742253] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11560.761685] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11566.880247] loop2: detected capacity change from 0 to 204800000 [11573.998744] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11574.133787] LustreError: fs25660-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11574.143605] LustreError: fs25660-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11574.163855] LustreError: 308702:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11574.168899] LustreError: 308702:0:(tgt_mount.c:2129:server_put_super()) no obd fs25660-OST0001 [11574.178585] LustreError: 308702:0:(tgt_mount.c:116:server_deregister_mount()) fs25660-OST0001 not registered [11574.213265] Lustre: server umount fs25660-OST0001 complete [11574.217231] LustreError: 308702:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11587.809135] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11587.898105] Lustre: fs25660-OST0001: new disk, initializing [11587.900995] Lustre: srv-fs25660-OST0001: No data found on store. Initialize space. [11587.905560] Lustre: 309253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11587.974395] Lustre: fs25660-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11595.764701] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11595.796229] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [11597.269259] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11606.594653] Lustre: server umount fs25660-OST0000 complete [11616.660317] Lustre: server umount fs25660-OST0001 complete [11628.956828] loop1: detected capacity change from 0 to 204800000 [11634.124857] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11634.261941] Lustre: fs25660-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11640.131827] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11648.480569] loop2: detected capacity change from 0 to 204800000 [11655.935884] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11664.681540] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11675.450490] Lustre: server umount fs25660-OST0000 complete [11685.877852] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11688.460117] Lustre: server umount fs25660-OST0001 complete [11690.989618] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11705.313097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11705.319890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11705.341728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11706.341391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.345897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11706.359558] Lustre: Skipped 1 previous similar message [11710.130638] Lustre: server umount lustre-OST0000 complete [11716.576723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.578447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11716.595952] Lustre: Skipped 1 previous similar message [11720.594795] Lustre: server umount lustre-MDT0000 complete [11725.210291] LustreError: 301896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757969 with bad export cookie 8028195178257790303 [11725.214876] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11725.223924] LustreError: 301896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11726.122939] Lustre: server umount lustre-MDT0001 complete [11738.022804] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [11742.665846] Key type lgssc unregistered [11743.179486] LNet: 315183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11743.193855] LNetError: 315183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11743.225597] LNet: Removed LNI 192.168.201.132@tcp [11744.944349] Key type .llcrypt unregistered [11744.946259] Key type ._llcrypt unregistered [11782.204258] Key type ._llcrypt registered [11782.206319] Key type .llcrypt registered [11782.349463] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [11797.888498] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [11799.290553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11799.526594] alg: No test for adler32 (adler32-zlib) [11800.568388] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [11800.770621] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [11802.464784] Key type lgssc registered [11803.570411] Lustre: Echo OBD driver; http://www.lustre.org/ [11843.969385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11843.993096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.296432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11845.322290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11845.425413] Lustre: lustre-MDT0000: new disk, initializing [11845.497508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11845.513570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11851.143695] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11864.062456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11864.141096] Lustre: 319051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11864.179786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11864.183036] Lustre: Skipped 1 previous similar message [11864.234969] Lustre: lustre-MDT0001: new disk, initializing [11864.314508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11864.345854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11864.367944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11868.733886] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11880.883963] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11889.503023] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11897.760375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11898.312392] Lustre: lustre-OST0000: new disk, initializing [11898.317249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11898.326370] Lustre: 321019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11898.413180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11906.365630] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11908.202574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11908.209593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11908.349704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11917.907314] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11920.199575] LustreError: 317224:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [11920.232186] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325 [11921.343428] Lustre: Unmounted lustre-client [11934.185214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11934.195980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11934.209083] LustreError: Skipped 1 previous similar message [11934.237635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11934.239670] Lustre: Skipped 1 previous similar message [11934.251812] Lustre: Skipped 1 previous similar message [11935.716735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11935.736329] Lustre: Skipped 1 previous similar message [11939.106774] Lustre: server umount lustre-OST0000 complete [11945.953910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.957082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.962968] Lustre: Skipped 1 previous similar message [11951.073375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11951.084258] Lustre: Skipped 2 previous similar messages [11956.194483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11956.203771] Lustre: Skipped 1 previous similar message [11957.215329] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11957.360764] Lustre: server umount lustre-MDT0000 complete [11961.215538] LustreError: 319041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758205 with bad export cookie 12987588491593686529 [11961.221973] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11961.230880] LustreError: 319041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11961.572446] Lustre: server umount lustre-MDT0001 complete [11973.041940] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [11975.518376] Key type lgssc unregistered [11983.008656] LNet: 323183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11983.025610] LNetError: 323183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12051.641790] LNet: Removed LNI 192.168.201.132@tcp [12052.454686] Key type .llcrypt unregistered [12052.457221] Key type ._llcrypt unregistered [12070.361763] Key type ._llcrypt registered [12070.364174] Key type .llcrypt registered [12070.520308] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [12088.611618] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [12090.097903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12090.119923] alg: No test for adler32 (adler32-zlib) [12091.225632] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [12091.446317] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [12093.111177] Key type lgssc registered [12094.273368] Lustre: Echo OBD driver; http://www.lustre.org/ [12127.556862] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:46:09 (1786758369) [12135.475585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12135.496851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.777839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12136.809726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12136.873445] Lustre: lustre-MDT0000: new disk, initializing [12136.947590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12136.963981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12140.606088] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12152.921228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12153.076917] Lustre: 326405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12153.188894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12153.199181] Lustre: Skipped 1 previous similar message [12153.320889] Lustre: lustre-MDT0001: new disk, initializing [12153.474804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12153.544652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12153.566788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12158.837438] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12169.734456] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12176.181869] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12185.177484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12185.343357] Lustre: lustre-OST0000: new disk, initializing [12185.346398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12185.351723] Lustre: 328374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12185.417775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12190.817640] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12194.340664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12194.352233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12194.467780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12200.719897] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12215.265550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12215.270250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12215.275040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12215.276415] Lustre: Skipped 1 previous similar message [12220.385825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12220.393639] Lustre: Skipped 2 previous similar messages [12221.208474] Lustre: server umount lustre-OST0000 complete [12224.994924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12225.019535] LustreError: Skipped 1 previous similar message [12225.021668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12225.050828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12230.624332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12230.630888] Lustre: Skipped 3 previous similar messages [12235.745269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12235.756848] Lustre: Skipped 1 previous similar message [12239.327233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12239.582897] Lustre: server umount lustre-MDT0000 complete [12243.066622] LustreError: 326398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758486 with bad export cookie 8295062714883113967 [12243.068378] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12243.074981] LustreError: 326398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12243.299610] Lustre: server umount lustre-MDT0001 complete [12253.707578] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12256.247394] Key type lgssc unregistered [12256.698436] LNet: 330287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12256.717713] LNetError: 330287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12256.751711] LNet: Removed LNI 192.168.201.132@tcp [12257.757439] Key type .llcrypt unregistered [12257.759326] Key type ._llcrypt unregistered [12277.856785] Key type ._llcrypt registered [12277.859753] Key type .llcrypt registered [12278.004673] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [12279.329786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12279.341478] alg: No test for adler32 (adler32-zlib) [12280.443716] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [12280.707955] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [12282.439464] Key type lgssc registered [12283.771148] Lustre: Echo OBD driver; http://www.lustre.org/ [12290.987744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12291.000900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.499898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.011611] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12302.817497] LustreError: 331562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.837390] LustreError: 331562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12304.462957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12304.784627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12309.256726] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12316.588097] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12322.214593] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12328.384535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12328.803054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12334.570512] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12342.304549] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12351.296310] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12352.753265] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12358.629290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12358.634932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12358.647988] Lustre: Skipped 2 previous similar messages [12358.656572] Lustre: Skipped 1 previous similar message [12359.418215] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [12359.424134] Lustre: Skipped 1 previous similar message [12362.672939] Lustre: server umount lustre-MDT0000 complete [12363.747785] LustreError: 332289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12363.779270] LustreError: 332289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12365.639043] LustreError: 331542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758609 with bad export cookie 13441936405651456561 [12365.650384] LustreError: 331542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12365.654351] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12368.866245] LustreError: 331558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12368.866975] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12368.878172] LustreError: 331558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12368.900603] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12370.914809] LustreError: 333402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12370.914852] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12370.933149] LustreError: 333402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12370.948680] Lustre: Skipped 1 previous similar message [12372.112715] Lustre: server umount lustre-MDT0001 complete [12375.772553] Lustre: server umount lustre-OST0000 complete [12384.220523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12388.521569] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12391.319250] Lustre: 335740:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.132@tcp [12393.364163] Lustre: server umount MGS complete [12401.946524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12402.230578] LustreError: 336314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12402.288495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12406.616834] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12412.906940] LustreError: 336315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.929146] LustreError: 336315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12415.831699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12416.202684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12420.378897] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12428.434690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12428.728578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12435.334850] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12442.359207] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12443.866308] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12448.872653] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12452.408730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [12458.465960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12458.477037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12458.483218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12458.494875] Lustre: Skipped 1 previous similar message [12463.024583] Lustre: server umount lustre-OST0000 complete [12467.694743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12467.698565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12467.709389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12467.712047] Lustre: Skipped 1 previous similar message [12472.585360] Lustre: server umount lustre-MDT0000 complete [12473.824784] LustreError: 338280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12473.833788] LustreError: 338280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12476.360216] LustreError: 336295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758720 with bad export cookie 13441936405651457401 [12476.368187] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12476.369695] LustreError: 336295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12476.672680] Lustre: server umount lustre-MDT0001 complete [12486.556721] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12489.165993] Key type lgssc unregistered [12489.399707] LNet: 339753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12489.408483] LNetError: 339753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12489.421143] LNet: Removed LNI 192.168.201.132@tcp [12490.092240] Key type .llcrypt unregistered [12490.093754] Key type ._llcrypt unregistered [12509.326632] Key type ._llcrypt registered [12509.329987] Key type .llcrypt registered [12509.422055] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [12524.756400] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [12525.636258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12525.658206] alg: No test for adler32 (adler32-zlib) [12526.788601] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [12527.022135] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [12528.703467] Key type lgssc registered [12529.842798] Lustre: Echo OBD driver; http://www.lustre.org/ [12568.634825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12568.656552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12569.877882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12569.919070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12570.035141] Lustre: lustre-MDT0000: new disk, initializing [12570.135880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12570.155919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12575.059979] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12586.760302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12586.887798] Lustre: 342913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12586.931155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12586.934588] Lustre: Skipped 1 previous similar message [12587.019936] Lustre: lustre-MDT0001: new disk, initializing [12587.160294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12587.216755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12587.236640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12592.031647] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12602.749884] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12609.078383] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12616.366984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12616.741900] Lustre: lustre-OST0000: new disk, initializing [12616.746588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12616.753984] Lustre: 344884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12616.888563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12622.999254] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12624.446945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12624.460982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12624.544598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12632.414661] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12636.985892] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12637.285287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12641.391419] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12641.521954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12645.347356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12645.350046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12645.354201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12649.291304] Lustre: server umount lustre-OST0000 complete [12655.584199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12655.592567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12655.597295] Lustre: Skipped 2 previous similar messages [12655.617306] Lustre: Skipped 2 previous similar messages [12660.706931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12660.714928] Lustre: Skipped 1 previous similar message [12661.095200] Lustre: server umount lustre-MDT0000 complete [12665.603513] LustreError: 344881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758909 with bad export cookie 671071820641301991 [12665.606687] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12665.609368] LustreError: 344881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12665.873180] Lustre: server umount lustre-MDT0001 complete [12675.887772] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:55:17 (1786758917) [12695.120978] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [12704.427628] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [12739.852636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12740.200869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12740.228980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12740.287739] Lustre: lustre-MDT0000: new disk, initializing [12740.355900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12740.373247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12744.289344] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12754.596695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12754.792360] Lustre: 349644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12754.901695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12754.905966] Lustre: Skipped 1 previous similar message [12755.031389] Lustre: lustre-MDT0001: new disk, initializing [12755.162511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12755.203235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12755.212326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12760.066938] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12768.342286] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12774.232701] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12779.598978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12779.817337] Lustre: lustre-OST0000: new disk, initializing [12779.820465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12779.834863] Lustre: 351612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12779.937983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12786.089432] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12788.257142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12788.264835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12788.358669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12796.333877] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12829.674777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12829.691895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.717134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12834.275800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12834.290320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12834.315303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12835.560393] Lustre: server umount lustre-OST0000 complete [12839.912124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12839.929201] Lustre: Skipped 1 previous similar message [12845.024587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12845.030170] Lustre: Skipped 4 previous similar messages [12845.652264] Lustre: server umount lustre-MDT0000 complete [12849.732372] LustreError: 349635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759093 with bad export cookie 671071820641302873 [12849.741910] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12849.744713] LustreError: 349635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12850.207509] Lustre: server umount lustre-MDT0001 complete [12859.192945] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12862.003718] Key type lgssc unregistered [12862.287602] LNet: 353788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12862.298710] LNetError: 353788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12862.313648] LNet: Removed LNI 192.168.201.132@tcp [12863.042240] Key type .llcrypt unregistered [12863.048242] Key type ._llcrypt unregistered [12965.639447] Key type ._llcrypt registered [12965.642565] Key type .llcrypt registered [12965.777966] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12966.092608] Key type .llcrypt unregistered [12966.097985] Key type ._llcrypt unregistered [13045.520324] Key type ._llcrypt registered [13045.525195] Key type .llcrypt registered [13045.659823] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13046.043340] Key type .llcrypt unregistered [13046.044735] Key type ._llcrypt unregistered [13080.151697] Key type ._llcrypt registered [13080.154371] Key type .llcrypt registered [13080.270188] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13080.690297] Key type .llcrypt unregistered [13080.693650] Key type ._llcrypt unregistered [13118.414773] Key type ._llcrypt registered [13118.418401] Key type .llcrypt registered [13118.562561] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13118.874512] Key type .llcrypt unregistered [13118.881690] Key type ._llcrypt unregistered [13206.214336] Key type ._llcrypt registered [13206.218070] Key type .llcrypt registered [13206.367582] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13206.698870] Key type .llcrypt unregistered [13206.702092] Key type ._llcrypt unregistered [13237.974516] Key type ._llcrypt registered [13237.978228] Key type .llcrypt registered [13238.057352] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13238.313776] Key type .llcrypt unregistered [13238.316740] Key type ._llcrypt unregistered [13266.471927] Key type ._llcrypt registered [13266.474484] Key type .llcrypt registered [13266.565285] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13266.831734] Key type .llcrypt unregistered [13266.835530] Key type ._llcrypt unregistered [13294.099891] Key type ._llcrypt registered [13294.103362] Key type .llcrypt registered [13294.239300] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13294.656623] Key type .llcrypt unregistered [13294.663238] Key type ._llcrypt unregistered [13326.328556] Key type ._llcrypt registered [13326.337567] Key type .llcrypt registered [13326.547990] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13326.921349] Key type .llcrypt unregistered [13326.923159] Key type ._llcrypt unregistered [13361.601559] Key type ._llcrypt registered [13361.604427] Key type .llcrypt registered [13361.704680] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13362.010393] Key type .llcrypt unregistered [13362.012665] Key type ._llcrypt unregistered [13427.042619] Key type ._llcrypt registered [13427.044301] Key type .llcrypt registered [13427.159238] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13427.520322] Key type .llcrypt unregistered [13427.522884] Key type ._llcrypt unregistered [13458.968925] Key type ._llcrypt registered [13458.972432] Key type .llcrypt registered [13459.148847] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13459.663854] Key type .llcrypt unregistered [13459.668987] Key type ._llcrypt unregistered [13494.232264] Key type ._llcrypt registered [13494.234261] Key type .llcrypt registered [13494.314545] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13494.780965] Key type .llcrypt unregistered [13494.788388] Key type ._llcrypt unregistered [13560.443045] Key type ._llcrypt registered [13560.447429] Key type .llcrypt registered [13560.561738] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [13579.099405] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [13580.217172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13580.266673] alg: No test for adler32 (adler32-zlib) [13581.221476] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [13581.391833] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [13583.023182] Key type lgssc registered [13584.048756] Lustre: Echo OBD driver; http://www.lustre.org/ [13622.924391] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13622.944845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13624.477489] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13624.585316] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13624.731338] Lustre: tst250fs-MDT0000: new disk, initializing [13624.887842] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13624.912520] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13630.010407] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13641.257036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13641.376371] Lustre: 374188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13641.434366] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13641.441730] Lustre: Skipped 1 previous similar message [13641.550827] Lustre: tst250fs-MDT0001: new disk, initializing [13641.639873] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13641.663898] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13641.680865] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13645.303335] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13649.383131] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13656.330175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13656.601591] Lustre: tst250fs-OST0000: new disk, initializing [13656.604745] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13656.616290] Lustre: 375818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13656.725174] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13658.047381] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13658.058180] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13658.226497] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13663.168475] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13672.965196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13673.093326] Lustre: tst250fs-OST0001: new disk, initializing [13673.096550] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13673.102312] Lustre: 376691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13673.179317] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13679.458469] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13682.237159] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13682.252437] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13682.324710] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13685.483346] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13688.049626] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13688.052853] Lustre: Skipped 2 previous similar messages [13713.394869] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13713.401851] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13713.423043] Lustre: Skipped 2 previous similar messages [13713.447801] Lustre: Skipped 2 previous similar messages [13715.046159] Lustre: server umount tst250fs-MDT0000 complete [13720.962037] LustreError: 376690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759964 with bad export cookie 11276880208899380936 [13720.966829] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13720.969567] LustreError: 376690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13721.232173] Lustre: server umount tst250fs-MDT0001 complete [13736.419314] Lustre: server umount tst250fs-OST0000 complete [13739.807184] Lustre: 372413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786759967/real 1786759967] req@ffff97ca1d7b8700 x1873553470327040/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786759983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13739.850423] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13739.875647] Lustre: Skipped 1 previous similar message [13741.821505] Lustre: server umount tst250fs-OST0001 complete [13766.508721] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [13774.447840] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [13805.986347] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:14:08 (1786760048) [13807.415492] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13822.787359] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [13833.587583] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [13868.727591] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13614 sec ========= 22:15:10 (1786760110) [13871.364171] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:15:12 (1786760112) === [13875.220707] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:15:16 (1786760116) === [13917.502047] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13920.428500] Key type lgssc unregistered [13920.995651] LNet: 385826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13921.009408] LNetError: 385826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13921.051779] LNet: Removed LNI 192.168.201.132@tcp [13922.070161] Key type .llcrypt unregistered [13922.072570] Key type ._llcrypt unregistered