[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1708508283 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002268] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.010035] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.012062] *** VALIDATE selinux *** [ 0.022681] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031137] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033017] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035116] *** VALIDATE tmpfs *** [ 0.037086] *** VALIDATE proc *** [ 0.039096] *** VALIDATE cgroup *** [ 0.040009] *** VALIDATE cgroup2 *** [ 0.042355] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044034] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.045010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.046030] Spectre V2 : User space: Vulnerable [ 0.047012] Speculative Store Bypass: Vulnerable [ 0.050622] debug: unmapping init [mem 0xffffffff88859000-0xffffffff88860fff] [ 0.053264] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.054899] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.055028] ... version: 2 [ 0.056019] ... bit width: 48 [ 0.057016] ... generic registers: 4 [ 0.058019] ... value mask: 0000ffffffffffff [ 0.059020] ... max period: 00007fffffffffff [ 0.060020] ... fixed-purpose events: 3 [ 0.061019] ... event mask: 000000070000000f [ 0.063364] rcu: Hierarchical SRCU implementation. [ 0.066970] smp: Bringing up secondary CPUs ... [ 0.067653] x86: Booting SMP configuration: [ 0.068033] .... node #0, CPUs: #1 #2 #3 [ 0.083083] smp: Brought up 1 node, 4 CPUs [ 0.085016] smpboot: Max logical packages: 1 [ 0.086017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124352] node 0 deferred pages initialised in 35ms [ 0.129151] devtmpfs: initialized [ 0.130674] x86/mm: Memory block size: 128MB [ 0.133397] gcov: version magic: 0x41383552 [ 0.135304] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137557] pinctrl core: initialized pinctrl subsystem [ 0.138459] [ 0.138965] ************************************************************* [ 0.139019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140017] ** ** [ 0.141017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142018] ** ** [ 0.143011] ** This means that this kernel is built to expose internal ** [ 0.144009] ** IOMMU data structures, which may compromise security on ** [ 0.145014] ** your system. ** [ 0.146014] ** ** [ 0.147011] ** If you see this message and you are not debugging the ** [ 0.148009] ** kernel, report this immediately to your vendor! ** [ 0.149009] ** ** [ 0.150014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151011] ************************************************************* [ 0.153333] NET: Registered protocol family 16 [ 0.155318] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159519] cpuidle: using governor menu [ 0.163376] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178000] PCI: Using configuration type 1 for base access [ 0.187698] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216369] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220442] cryptd: max_cpu_qlen set to 1000 [ 0.222504] ACPI: Added _OSI(Module Device) [ 0.228017] ACPI: Added _OSI(Processor Device) [ 0.233021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.238017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256096] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.278138] ACPI: Interpreter enabled [ 0.279149] ACPI: PM: (supports S0 S3 S4 S5) [ 0.280015] ACPI: Using IOAPIC for interrupt routing [ 0.281126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.282444] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.298673] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.299058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.300022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.301082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.303515] acpiphp: Slot [2] registered [ 0.304185] acpiphp: Slot [5] registered [ 0.305166] acpiphp: Slot [6] registered [ 0.306143] acpiphp: Slot [7] registered [ 0.307200] acpiphp: Slot [8] registered [ 0.308138] acpiphp: Slot [9] registered [ 0.309139] acpiphp: Slot [10] registered [ 0.310239] acpiphp: Slot [3] registered [ 0.311132] acpiphp: Slot [4] registered [ 0.312115] acpiphp: Slot [11] registered [ 0.313163] acpiphp: Slot [12] registered [ 0.314113] acpiphp: Slot [13] registered [ 0.315134] acpiphp: Slot [14] registered [ 0.316101] acpiphp: Slot [15] registered [ 0.317134] acpiphp: Slot [16] registered [ 0.318106] acpiphp: Slot [17] registered [ 0.319118] acpiphp: Slot [18] registered [ 0.320106] acpiphp: Slot [19] registered [ 0.321174] acpiphp: Slot [20] registered [ 0.322103] acpiphp: Slot [21] registered [ 0.323113] acpiphp: Slot [22] registered [ 0.324266] acpiphp: Slot [23] registered [ 0.325096] acpiphp: Slot [24] registered [ 0.326115] acpiphp: Slot [25] registered [ 0.327109] acpiphp: Slot [26] registered [ 0.328097] acpiphp: Slot [27] registered [ 0.329085] acpiphp: Slot [28] registered [ 0.330131] acpiphp: Slot [29] registered [ 0.331101] acpiphp: Slot [30] registered [ 0.332139] acpiphp: Slot [31] registered [ 0.333053] PCI host bridge to bus 0000:00 [ 0.334020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342096] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344430] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350623] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.357020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358570] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359788] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.361875] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.364016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.378979] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.392050] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.399258] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.402018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.405018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.412029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.423401] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.426020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.429019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.436069] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.503142] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.510018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.518016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.537017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.548519] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.558019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.566020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.581015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.590235] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.596017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.604015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.631021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.652934] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.660849] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.664780] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.669618] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.672405] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.679060] iommu: Default domain type: Passthrough [ 0.680734] SCSI subsystem initialized [ 0.681263] ACPI: bus type USB registered [ 0.682114] usbcore: registered new interface driver usbfs [ 0.683089] usbcore: registered new interface driver hub [ 0.684169] usbcore: registered new device driver usb [ 0.685248] pps_core: LinuxPPS API ver. 1 registered [ 0.686011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.687114] PTP clock support registered [ 0.688175] EDAC MC: Ver: 3.0.0 [ 0.690099] PCI: Using ACPI for IRQ routing [ 0.692127] NetLabel: Initializing [ 0.694066] NetLabel: domain hash size = 128 [ 0.695009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.698083] NetLabel: unlabeled traffic allowed by default [ 0.700352] vgaarb: loaded [ 0.702061] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713221] clocksource: Switched to clocksource kvm-clock [ 0.949965] VFS: Disk quotas dquot_6.6.0 [ 0.962433] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.990368] *** VALIDATE ramfs *** [ 0.999868] *** VALIDATE hugetlbfs *** [ 1.203409] pnp: PnP ACPI init [ 1.210887] pnp: PnP ACPI: found 6 devices [ 1.528677] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.544448] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.556120] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.567809] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.580648] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.591343] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.605124] NET: Registered protocol family 2 [ 1.618656] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.632894] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.636942] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.643169] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.646936] TCP: Hash tables configured (established 65536 bind 65536) [ 1.649959] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.653609] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.656811] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.660245] NET: Registered protocol family 1 [ 1.949373] RPC: Registered named UNIX socket transport module. [ 1.951746] RPC: Registered udp transport module. [ 1.953698] RPC: Registered tcp transport module. [ 1.955329] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.957926] NET: Registered protocol family 44 [ 1.959787] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.961977] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.964342] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.966834] PCI: CLS 0 bytes, default 64 [ 1.968642] Unpacking initramfs... [ 5.021807] debug: unmapping init [mem 0xffffa0c5bcc54000-0xffffa0c5bffbffff] [ 5.026244] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.028853] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.032057] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.545450] Initialise system trusted keyrings [ 8.549729] Key type blacklist registered [ 8.554977] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.864746] zbud: loaded [ 9.027338] *** VALIDATE nfs *** [ 9.032194] *** VALIDATE nfs4 *** [ 9.034161] pstore: using deflate compression [ 9.037511] Platform Keyring initialized [ 9.140513] NET: Registered protocol family 38 [ 9.142421] Key type asymmetric registered [ 9.143823] Asymmetric key parser 'x509' registered [ 9.145456] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 9.152350] io scheduler mq-deadline registered [ 9.154339] io scheduler kyber registered [ 9.156433] io scheduler bfq registered [ 9.158333] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 9.161318] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 9.163989] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 9.166788] ACPI: Power Button [PWRF] [ 9.172645] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.180462] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.227092] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.253135] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.324805] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.359961] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.387267] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.392362] Non-volatile memory driver v1.3 [ 9.394302] Linux agpgart interface v0.103 [ 9.424486] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 9.427507] vda: detected capacity change from 0 to 74645504 [ 9.486636] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.490261] vdb: detected capacity change from 0 to 1073741824 [ 9.516582] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.519840] vdc: detected capacity change from 0 to 2621440000 [ 9.537930] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.541127] vdd: detected capacity change from 0 to 2621440000 [ 9.556620] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.559631] vde: detected capacity change from 0 to 4294967296 [ 9.574976] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.578248] vdf: detected capacity change from 0 to 4294967296 [ 9.585395] libphy: Fixed MDIO Bus: probed [ 9.599421] usbcore: registered new interface driver usbserial_generic [ 9.602203] usbserial: USB Serial support registered for generic [ 9.604700] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.609988] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.612203] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.614872] mousedev: PS/2 mouse device common for all mice [ 9.620635] rtc_cmos 00:05: RTC can wake from S4 [ 9.623981] rtc_cmos 00:05: registered as rtc0 [ 9.626265] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.629993] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.634283] intel_pstate: CPU model not supported [ 9.637602] hid: raw HID events driver (C) Jiri Kosina [ 9.640777] usbcore: registered new interface driver usbhid [ 9.643213] usbhid: USB HID core driver [ 9.645480] drop_monitor: Initializing network drop monitor service [ 9.646808] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.650364] Initializing XFRM netlink socket [ 9.659637] NET: Registered protocol family 10 [ 9.662945] Segment Routing with IPv6 [ 9.664577] NET: Registered protocol family 17 [ 9.666965] mpls_gso: MPLS GSO support [ 9.672127] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.677836] RAS: Correctable Errors collector initialized. [ 9.679859] AVX version of gcm_enc/dec engaged. [ 9.681680] AES CTR mode by8 optimization enabled [ 9.771772] sched_clock: Marking stable (9771750949, 0)->(12256435433, -2484684484) [ 9.775797] registered taskstats version 1 [ 9.778250] Loading compiled-in X.509 certificates [ 9.780320] zswap: loaded using pool lzo/zbud [ 9.813751] Key type big_key registered [ 9.828033] Key type encrypted registered [ 9.829751] ima: No TPM chip found, activating TPM-bypass! [ 9.832442] ima: Allocated hash algorithm: sha1 [ 9.834355] ima: No architecture policies found [ 9.836241] evm: Initialising EVM extended attributes: [ 9.838411] evm: security.selinux [ 9.839710] evm: security.ima [ 9.840887] evm: security.capability [ 9.842437] evm: HMAC attrs: 0x1 [ 9.844916] rtc_cmos 00:05: setting system clock to 2026-08-18 05:44:08 UTC (1787031848) [ 9.852194] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 9.855729] debug: unmapping init [mem 0xffffffff88582000-0xffffffff88858fff] [ 9.865297] Write protecting the kernel read-only data: 28672k [ 9.870366] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 9.928365] debug: unmapping init [mem 0xffffffff87514000-0xffffffff875fffff] [ 10.000266] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.018457] systemd[1]: Detected virtualization kvm. [ 10.023505] systemd[1]: Detected architecture x86-64. [ 10.026905] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.063936] systemd[1]: No hostname configured. [ 10.066826] systemd[1]: Set hostname to . [ 10.071425] random: systemd: uninitialized urandom read (16 bytes read) [ 10.075260] systemd[1]: Initializing machine ID from random generator. [ 11.068939] random: systemd: uninitialized urandom read (16 bytes read) [ 11.075823] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 11.084757] random: systemd: uninitialized urandom read (16 bytes read) [ 11.094927] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 11.107270] random: systemd: uninitialized urandom read (16 bytes read) [ 11.114281] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 11.126322] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 14.712584] device-mapper: uevent: version 1.0.3 [ 14.719561] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 18.057449] virtio_net virtio0 ens2: renamed from eth0 [ 18.185107] random: fast init done [ 18.570322] scsi host0: ata_piix [ 18.607265] scsi host1: ata_piix [ 18.608733] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 18.611819] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 23.361508] random: crng init done [ 23.363679] random: 5 urandom warning(s) missed due to ratelimiting [ 27.098809] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 29.615697] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 34.588281] printk: systemd: 26 output lines suppressed due to ratelimiting [ 36.177015] hrtimer: interrupt took 6922999 ns [ 36.253707] SELinux: Disabled at runtime. [ 36.470211] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 36.500410] systemd[1]: Detected virtualization kvm. [ 36.502349] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 39.383634] systemd[1]: initrd-switch-root.service: Succeeded. [ 39.398241] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 39.441599] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 39.462241] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 39.475477] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 39.516771] systemd[1]: Starting Journal Service... Starting Journal Service... [ 39.602196] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 39.925147] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 42.412807] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 44.051447] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 44.308578] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 45.472840] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 45.753584] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 52.129436] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit)[ 53.636493] NFS: Registering the id_resolver key type [ 53.640741] Key type id_resolver registered [ 53.652731] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg137-server login: [ 129.835145] libcfs: loading out-of-tree module taints kernel. [ 129.926535] Key type ._llcrypt registered [ 129.932610] Key type .llcrypt registered [ 130.123331] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 150.878786] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 152.799814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 152.853719] alg: No test for adler32 (adler32-zlib) [ 154.529175] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 156.018646] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 157.847580] Key type lgssc registered [ 159.398967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 177.480318] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 220.472844] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 232.934841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 232.983219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.224375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 234.246928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 234.396133] Lustre: lustre-MDT0000: new disk, initializing [ 234.463440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.482350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 238.704516] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.620375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.732977] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 250.763440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 250.767398] Lustre: Skipped 1 previous similar message [ 250.865415] Lustre: lustre-MDT0001: new disk, initializing [ 250.926190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.947075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 250.959774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 255.449946] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.232317] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 269.502451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.891700] Lustre: lustre-OST0000: new disk, initializing [ 269.898971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 269.907921] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.977756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 275.906489] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.524069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 276.536135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 276.647163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 290.058338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 290.261139] Lustre: lustre-OST0001: new disk, initializing [ 290.271462] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 290.283760] Lustre: 9529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 290.403033] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 298.102251] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 298.132772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 298.166511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 298.266794] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 312.867906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 320.438216] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 326.485786] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing check_logdir /tmp/testlogs/ [ 331.189992] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing yml_node [ 334.811992] Lustre: DEBUG MARKER: Client: 2.17.55.1 [ 337.483167] Lustre: DEBUG MARKER: MDS: 2.17.55.1 [ 340.258460] Lustre: DEBUG MARKER: OSS: 2.17.55.1 [ 341.922304] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 18 01:49:39 EDT 2026 [ 360.675623] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 362.408916] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 378.848112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.870191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.888839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.387067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.392837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.407306] Lustre: Skipped 2 previous similar messages [ 380.424036] Lustre: Skipped 3 previous similar messages [ 384.850459] Lustre: server umount lustre-MDT0000 complete [ 388.621703] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787032227 with bad export cookie 16180595030041633484 [ 388.622516] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 388.640190] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 390.624417] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.624928] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.625756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.649626] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 390.704248] Lustre: Skipped 1 previous similar message [ 393.642269] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.642778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 393.669524] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 393.687047] Lustre: Skipped 2 previous similar messages [ 395.605964] Lustre: server umount lustre-MDT0001 complete [ 400.428984] Lustre: server umount lustre-OST0000 complete [ 405.486110] Lustre: server umount lustre-OST0001 complete [ 414.044748] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 424.104583] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 462.626425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.890114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 462.963651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 463.054736] Lustre: lustre-MDT0000: new disk, initializing [ 463.165971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.187050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 468.195803] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 480.815305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.063989] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 481.200238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 481.207204] Lustre: Skipped 1 previous similar message [ 481.363072] Lustre: lustre-MDT0001: new disk, initializing [ 481.566726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 481.695139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 481.706876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 488.085794] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 498.436846] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 504.860944] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 511.906145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.288711] Lustre: lustre-OST0000: new disk, initializing [ 512.292464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 512.300640] Lustre: 17051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 512.446719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.045545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 514.057793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 514.198584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 519.539082] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 530.208219] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 535.885706] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 536.166972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 541.187610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 541.367679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 544.736434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.745819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.754390] LustreError: Skipped 1 previous similar message [ 544.755125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.800612] Lustre: Skipped 1 previous similar message [ 549.873091] Lustre: server umount lustre-OST0000 complete [ 554.981672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.989612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 555.020992] Lustre: Skipped 1 previous similar message [ 555.038991] Lustre: Skipped 2 previous similar messages [ 560.094349] Lustre: server umount lustre-MDT0000 complete [ 560.111303] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.131431] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 563.751240] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787032402 with bad export cookie 16180595030041635017 [ 563.760932] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.769047] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 564.155066] Lustre: server umount lustre-MDT0001 complete [ 570.848846] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:53:28 (1787032408) [ 591.743025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.581618] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.732829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.389459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 597.987300] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.931672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.034186] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 622.197934] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 628.274705] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 630.581579] Lustre: Failing over lustre-MDT0000 [ 630.871619] Lustre: server umount lustre-MDT0000 complete [ 632.680603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.683925] LustreError: 20900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.693915] Lustre: Skipped 1 previous similar message [ 632.722980] LustreError: 20900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 648.863167] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787032471/real 1787032471] req@ffffa0c502040380 x1873838860299520/t0(0) o400->MGC192.168.201.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787032487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 648.880871] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 649.445174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.025637] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.041309] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 659.432618] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe08d05738c152e4f [ 659.460070] Lustre: MGC192.168.201.137@tcp: Connection restored to 0@lo (at 0@lo) [ 659.674232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.682937] Lustre: Skipped 1 previous similar message [ 659.732847] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 664.532486] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 665.063162] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 665.080142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 665.121821] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 672.389745] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 678.177271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.181688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 678.189344] Lustre: Skipped 1 previous similar message [ 682.531532] Lustre: server umount lustre-MDT0000 complete [ 686.149756] LustreError: 21497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787032524 with bad export cookie 16180595030041636431 [ 686.151269] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 686.159028] LustreError: 21497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 686.744945] Lustre: server umount lustre-MDT0001 complete [ 694.624326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.046275] LustreError: 23268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.068212] LustreError: 23268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 699.550268] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 707.252743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.972989] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 718.224434] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 724.148148] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 726.189914] Lustre: Failing over lustre-MDT0000 [ 726.428768] Lustre: server umount lustre-MDT0000 complete [ 728.037909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 728.049920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.735531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.892511] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.123609] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 748.951293] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 750.565618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 750.572591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 750.589478] Lustre: Skipped 1 previous similar message [ 750.621450] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 757.410912] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 764.907874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 764.915957] Lustre: Skipped 2 previous similar messages [ 767.711573] Lustre: server umount lustre-MDT0000 complete [ 770.016771] LustreError: 23268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.044262] LustreError: 23268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 771.248945] LustreError: 23248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787032609 with bad export cookie 16180595030041637369 [ 771.252050] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.259639] LustreError: 23248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 771.822918] Lustre: server umount lustre-MDT0001 complete [ 781.110911] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:56:58 (1787032618) [ 797.898615] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 808.326696] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 846.334819] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 857.257673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.521311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 857.557340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 857.628438] Lustre: lustre-MDT0000: new disk, initializing [ 857.700678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.705286] Lustre: Skipped 3 previous similar messages [ 857.718207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 861.845800] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 873.155403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.294039] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 873.627861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 878.039104] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 888.436799] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 895.342518] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 903.961233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 904.281081] Lustre: 31610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 906.009152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 906.072408] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 906.128561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 906.144723] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 911.381422] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 921.813928] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 927.127292] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 947.172763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.173820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.174702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.184919] Lustre: Skipped 4 previous similar messages [ 952.044225] Lustre: server umount lustre-OST0000 complete [ 970.207639] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 971.747358] LustreError: 31618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.763343] LustreError: 31618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 972.445151] Lustre: server umount lustre-MDT0000 complete [ 977.423075] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787032816 with bad export cookie 16180595030041637775 [ 977.439378] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.443258] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 977.752415] Lustre: server umount lustre-MDT0001 complete [ 989.126761] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 992.243666] Key type lgssc unregistered [ 992.652131] LNet: 33589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.658258] LNetError: 33589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.707208] LNet: Removed LNI 192.168.201.137@tcp [ 995.090520] Key type .llcrypt unregistered [ 995.095917] Key type ._llcrypt unregistered [ 1019.967626] Key type ._llcrypt registered [ 1019.973427] Key type .llcrypt registered [ 1020.073680] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1021.376193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.406037] alg: No test for adler32 (adler32-zlib) [ 1022.417625] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 1022.632902] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1024.450127] Key type lgssc registered [ 1025.471616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1033.029747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1033.056791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.420245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.022226] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1044.968484] LustreError: 35019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.978905] LustreError: 35019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1047.347376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.832638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1052.163730] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1059.106199] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1065.268856] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1073.562715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1073.793396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.607898] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1085.447982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1087.647869] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.294392] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1103.295553] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:02:21 (1787032941) [ 1105.460276] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1107.136470] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1113.310814] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:02:30 (1787032950) [ 1122.118620] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:02:39 (1787032959) [ 1123.186246] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1132.055792] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1132.064083] Lustre: Skipped 2 previous similar messages [ 1133.972212] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:02:51 (1787032971) [ 1136.254200] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1136.262859] Lustre: Skipped 2 previous similar messages [ 1145.376423] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1145.382145] Lustre: Skipped 4 previous similar messages [ 1159.714198] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:03:17 (1787032997) [ 1162.569614] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1165.873966] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1173.943732] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:03:31 (1787033011) [ 1175.683440] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1175.687925] Lustre: Skipped 6 previous similar messages [ 1188.679741] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:03:46 (1787033026) [ 1206.253413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.279899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1206.285142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1206.304454] Lustre: Skipped 1 previous similar message [ 1210.516260] Lustre: server umount lustre-OST0000 complete [ 1216.481582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.516239] Lustre: Skipped 1 previous similar message [ 1216.528955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1216.543657] Lustre: Skipped 1 previous similar message [ 1221.604929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1221.612422] Lustre: Skipped 1 previous similar message [ 1226.724579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.731526] Lustre: Skipped 1 previous similar message [ 1229.279185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.596870] Lustre: server umount lustre-MDT0000 complete [ 1231.841354] LustreError: 37510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.876954] LustreError: 37510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1233.764728] LustreError: 37939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787033072 with bad export cookie 7082415861788721398 [ 1233.783961] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.784320] LustreError: 37939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1234.078338] Lustre: server umount lustre-MDT0001 complete [ 1246.196371] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1249.177814] Key type lgssc unregistered [ 1249.514770] LNet: 42876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.523228] LNetError: 42876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.547379] LNet: Removed LNI 192.168.201.137@tcp [ 1250.289880] Key type .llcrypt unregistered [ 1250.292365] Key type ._llcrypt unregistered [ 1267.439880] Key type ._llcrypt registered [ 1267.442190] Key type .llcrypt registered [ 1267.534357] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 1281.655242] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1282.341038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1282.371391] alg: No test for adler32 (adler32-zlib) [ 1283.456834] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 1283.635787] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1285.295172] Key type lgssc registered [ 1286.219448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1317.056710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1317.122159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.361416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1318.386994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1318.516874] Lustre: lustre-MDT0000: new disk, initializing [ 1318.623341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.638446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1323.516574] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1334.955297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.062926] Lustre: 46029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1335.115777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1335.119310] Lustre: Skipped 1 previous similar message [ 1335.182627] Lustre: lustre-MDT0001: new disk, initializing [ 1335.285575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.321821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1335.335500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1339.877146] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1349.767567] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1355.698552] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1363.102431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1363.416444] Lustre: lustre-OST0000: new disk, initializing [ 1363.423240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1363.432734] Lustre: 47997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1363.568303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.654722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1369.668067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1369.810615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1371.133035] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1380.325697] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1384.583242] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1384.769225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1389.424580] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1389.594074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1395.697427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.698146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1395.700204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1395.713689] Lustre: Skipped 1 previous similar message [ 1395.751321] LustreError: Skipped 1 previous similar message [ 1397.665197] Lustre: server umount lustre-OST0000 complete [ 1401.824872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.828483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.835505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.838062] Lustre: Skipped 1 previous similar message [ 1405.921088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.931752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1405.950650] Lustre: Skipped 1 previous similar message [ 1407.862828] Lustre: server umount lustre-MDT0000 complete [ 1411.124782] LustreError: 46021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787033249 with bad export cookie 1493284469065269246 [ 1411.139618] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1411.146554] LustreError: 46021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1411.523380] Lustre: server umount lustre-MDT0001 complete [ 1421.265181] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:07:38 (1787033258) [ 1429.811272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.489664] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.513681] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1430.567680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.627419] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.392916] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1440.736504] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.857775] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.976654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.457212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1450.999364] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1458.022135] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1466.867517] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1473.907167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1474.258020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.927492] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1486.911535] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.659218] LustreError: 52986:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1495.727610] Lustre: Modifying parameter general.timeout=1 in log params [ 1496.729366] Lustre: Modifying parameter general.timeout=2 in log params [ 1499.011944] Lustre: Modifying parameter general.timeout=5 in log params [ 1499.016116] Lustre: Skipped 2 previous similar messages [ 1503.142252] Lustre: Modifying parameter general.timeout=11 in log params [ 1503.152599] Lustre: Skipped 5 previous similar messages [ 1511.208597] Lustre: Modifying parameter general.timeout=21 in log params [ 1511.211953] Lustre: Skipped 9 previous similar messages [ 1528.158099] Lustre: Modifying parameter general.timeout=41 in log params [ 1528.162234] Lustre: Skipped 19 previous similar messages [ 1560.741633] Lustre: Modifying parameter general.timeout=77 in log params [ 1560.755133] Lustre: Skipped 35 previous similar messages [ 1610.323533] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:10:48 (1787033448) [ 1612.710384] Lustre: 59924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1614.547932] Lustre: 60035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1618.321950] Lustre: 60234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1618.331526] Lustre: 60234:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1625.156530] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1625.160643] Lustre: Skipped 61 previous similar messages [ 1645.479156] Lustre: 61802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1645.492541] Lustre: 61802:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1645.519655] LustreError: 61802:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1649.861587] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:11:27 (1787033487) [ 1664.994540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.999757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.025070] Lustre: Skipped 2 previous similar messages [ 1665.040735] Lustre: Skipped 2 previous similar messages [ 1668.027438] Lustre: server umount lustre-MDT0000 complete [ 1670.113602] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.137246] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1671.827799] LustreError: 53035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787033510 with bad export cookie 1493284469065270128 [ 1671.839802] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1671.847633] LustreError: 53035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1672.150588] Lustre: server umount lustre-MDT0001 complete [ 1685.661405] Lustre: server umount lustre-OST0000 complete [ 1709.227604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.482699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1709.810129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.976174] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1721.711731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.873247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1721.894643] Lustre: 64335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1725.836469] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1732.270767] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1737.084048] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1743.018063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1743.152726] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1743.304298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.312942] Lustre: Skipped 1 previous similar message [ 1747.924279] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1753.833324] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1755.330310] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1755.338560] Lustre: Skipped 14 previous similar messages [ 1755.394705] Lustre: 66897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1761.766975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.773665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1761.775994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.625664] Lustre: server umount lustre-OST0000 complete [ 1772.002061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.014136] Lustre: Skipped 1 previous similar message [ 1772.026306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1772.039168] Lustre: Skipped 3 previous similar messages [ 1777.458174] Lustre: server umount lustre-MDT0000 complete [ 1781.166992] LustreError: 64326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787033619 with bad export cookie 1493284469065273089 [ 1781.171471] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1781.174179] LustreError: 64326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1781.492549] Lustre: server umount lustre-MDT0001 complete [ 1790.684453] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1792.965976] Key type lgssc unregistered [ 1793.269890] LNet: 68109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.273943] LNetError: 68109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.287733] LNet: Removed LNI 192.168.201.137@tcp [ 1794.164300] Key type .llcrypt unregistered [ 1794.168058] Key type ._llcrypt unregistered [ 1814.798737] Key type ._llcrypt registered [ 1814.807300] Key type .llcrypt registered [ 1814.898965] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1816.281246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.301162] alg: No test for adler32 (adler32-zlib) [ 1817.373575] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 1817.631619] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1819.321384] Key type lgssc registered [ 1820.406960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.996206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.029921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.635079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.552465] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1839.076575] LustreError: 69504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.087337] LustreError: 69504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1841.881473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.374859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1845.880155] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1851.960783] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1856.620573] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1862.586992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.177424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.434423] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1875.500427] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1882.960988] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1893.348648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1893.352847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.366161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.861281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.876379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.885408] Lustre: Skipped 1 previous similar message [ 1895.604489] Lustre: server umount lustre-MDT0000 complete [ 1898.920040] LustreError: 69484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787033737 with bad export cookie 11261623462661564865 [ 1898.935276] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.975808] LustreError: 71842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.984020] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.998823] Lustre: Skipped 1 previous similar message [ 1899.002772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1899.181826] Lustre: server umount lustre-MDT0001 complete [ 1909.031281] Lustre: server umount lustre-OST0000 complete [ 1933.661377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.896555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1933.939971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1934.291872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.090187] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1946.517161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.607730] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1946.636326] Lustre: 74381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1946.991219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.331811] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1956.948368] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1961.648950] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.117630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.271121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1967.589731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.971537] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1978.353737] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1979.879465] Lustre: 76954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1981.334975] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1986.532483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.532486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.533630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.538217] LustreError: Skipped 1 previous similar message [ 1986.559073] Lustre: Skipped 1 previous similar message [ 1991.515893] Lustre: server umount lustre-OST0000 complete [ 1996.776718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.779171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.792730] Lustre: Skipped 2 previous similar messages [ 1996.808105] Lustre: Skipped 1 previous similar message [ 2001.232659] Lustre: server umount lustre-MDT0000 complete [ 2001.889456] LustreError: 74391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.905409] LustreError: 74391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2004.608791] LustreError: 76956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787033843 with bad export cookie 11261623462661565698 [ 2004.616349] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.619374] LustreError: 76956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2004.915442] Lustre: server umount lustre-MDT0001 complete [ 2013.867570] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2016.022320] Key type lgssc unregistered [ 2016.292508] LNet: 78161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.300579] LNetError: 78161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.314336] LNet: Removed LNI 192.168.201.137@tcp [ 2017.126528] Key type .llcrypt unregistered [ 2017.129039] Key type ._llcrypt unregistered [ 2038.771284] Key type ._llcrypt registered [ 2038.773297] Key type .llcrypt registered [ 2038.862514] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2039.693793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.787249] alg: No test for adler32 (adler32-zlib) [ 2040.862209] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 2041.104678] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2042.783526] Key type lgssc registered [ 2043.776588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.215307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2050.236050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.641160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.624836] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2061.307044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.700400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.235397] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2071.577327] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2076.729585] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.682557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.080804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.714645] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2092.325884] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2102.101794] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2102.853622] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2104.214601] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2104.219205] Lustre: Skipped 1 previous similar message [ 2106.806649] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2106.811810] Lustre: Skipped 3 previous similar messages [ 2111.021354] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2111.027826] Lustre: Skipped 6 previous similar messages [ 2119.678200] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2119.682294] Lustre: Skipped 13 previous similar messages [ 2135.766275] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2135.773795] Lustre: Skipped 20 previous similar messages [ 2168.196400] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2168.200148] Lustre: Skipped 47 previous similar messages [ 2175.347537] Lustre: 87660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2177.288377] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:20:15 (1787034015) [ 2178.434246] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2179.634852] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:20:17 (1787034017) [ 2184.678827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.700864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2184.707873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.713844] Lustre: Skipped 1 previous similar message [ 2184.752221] LustreError: Skipped 1 previous similar message [ 2189.795159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.805075] Lustre: Skipped 2 previous similar messages [ 2190.745767] Lustre: server umount lustre-OST0000 complete [ 2194.914311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.933641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.934506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.953054] Lustre: Skipped 1 previous similar message [ 2200.033965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.060122] Lustre: Skipped 2 previous similar messages [ 2200.901770] Lustre: server umount lustre-MDT0000 complete [ 2205.154149] LustreError: 79571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.181893] LustreError: 79571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2205.885687] LustreError: 79555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787034044 with bad export cookie 11698016581232241488 [ 2205.903786] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.914205] LustreError: 79555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2206.758231] Lustre: server umount lustre-MDT0001 complete [ 2217.223608] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2220.568869] Key type lgssc unregistered [ 2220.911639] LNet: 88909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.917401] LNetError: 88909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.931665] LNet: Removed LNI 192.168.201.137@tcp [ 2222.997039] Key type .llcrypt unregistered [ 2222.999576] Key type ._llcrypt unregistered [ 2227.360804] Key type ._llcrypt registered [ 2227.363714] Key type .llcrypt registered [ 2227.459787] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2232.021716] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules [ 2232.711544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.729976] alg: No test for adler32 (adler32-zlib) [ 2233.910530] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 2234.126341] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2235.793067] Key type lgssc registered [ 2237.400340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2238.698134] LustreError: 90111:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2255.773356] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2260.241041] LustreError: 90111:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2264.673505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2264.689067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.320054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.320565] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2275.814527] LustreError: 91135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.848910] LustreError: 91135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2279.985421] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:21:57 (1787034117) [ 2280.931822] LustreError: 91136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.729337] LustreError: 91167:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2284.738635] LustreError: 91167:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2284.745254] LustreError: 91167:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2284.985318] Lustre: server umount lustre-MDT0000 complete [ 2297.091694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2299.268517] Key type lgssc unregistered [ 2299.494202] LNet: 92660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.502669] LNetError: 92660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.534188] LNet: Removed LNI 192.168.201.137@tcp [ 2301.217371] Key type .llcrypt unregistered [ 2301.220338] Key type ._llcrypt unregistered [ 2317.180777] Key type ._llcrypt registered [ 2317.188716] Key type .llcrypt registered [ 2317.297673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2318.611432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.646610] alg: No test for adler32 (adler32-zlib) [ 2319.619475] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 2319.824117] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2321.545990] Key type lgssc registered [ 2323.256402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2331.160525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2331.178460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.737357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.762729] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2342.881457] LustreError: 93878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.896499] LustreError: 93878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2343.751270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.233523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2347.791991] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2354.834655] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2360.150856] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2366.586994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.208577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.133485] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2380.192403] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2426.537347] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:24:24 (1787034264) [ 2428.303868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2430.578912] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:24:27 (1787034267) [ 2436.072116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2436.078314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.096354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.135839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.193793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.204031] Lustre: Skipped 1 previous similar message [ 2440.771470] Lustre: server umount lustre-MDT0000 complete [ 2444.270273] LustreError: 93874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.303228] LustreError: 93874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2444.321040] LustreError: 93858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787034282 with bad export cookie 2458608171122700696 [ 2444.325756] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.329534] LustreError: 93858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2444.773158] Lustre: server umount lustre-MDT0001 complete [ 2457.954757] Lustre: server umount lustre-OST0000 complete [ 2469.808971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.213030] LustreError: 98176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.481846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.490328] LustreError: 98177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.602725] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2479.600208] LustreError: 98176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.547075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.298638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.278737] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2499.661307] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2505.670500] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.517637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.810673] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2519.819408] LustreError: 98164:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2519.843398] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2519.849564] LustreError: 100259:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2519.854195] LustreError: 100259:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2519.858147] LustreError: 100259:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2519.918549] Lustre: server umount lustre-OST0000 complete [ 2519.921395] LustreError: 100259:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2528.181660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2528.476880] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2528.482314] LustreError: 98166:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2528.494692] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2528.507041] LustreError: 100719:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2528.517540] LustreError: 100719:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2528.526295] LustreError: 100719:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2528.605707] Lustre: server umount lustre-OST0000 complete [ 2528.609550] LustreError: 100719:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2537.502255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2537.744923] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2537.748931] Lustre: Found index 0 for lustre-OST0000, updating log [ 2537.755321] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2537.837451] Lustre: lustre-OST0000: new disk, initializing [ 2537.840908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2537.850962] Lustre: 101257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2538.110962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.451272] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2547.931753] Lustre: Failing over lustre-OST0000 [ 2548.101994] Lustre: server umount lustre-OST0000 complete [ 2553.321175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2553.334810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.353112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.357583] Lustre: Skipped 2 previous similar messages [ 2555.876859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2555.878150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.016457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.028867] Lustre: Skipped 4 previous similar messages [ 2563.143968] Lustre: server umount lustre-MDT0000 complete [ 2567.137510] LustreError: 98177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.154128] LustreError: 98177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2567.360319] LustreError: 101256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787034406 with bad export cookie 2458608171122703440 [ 2567.361671] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2567.368977] LustreError: 101256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2567.833157] Lustre: server umount lustre-MDT0001 complete [ 2578.347925] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:26:55 (1787034415) [ 2602.761699] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2617.053372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.559036] LustreError: 103622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.687179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.331687] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2631.206245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.253268] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2637.898691] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2644.010888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.635248] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2650.996293] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2656.888043] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2656.900299] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2656.930592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2660.451145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2660.520874] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2660.638795] Lustre: lustre-OST0001: new disk, initializing [ 2660.645608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2660.665964] Lustre: 105940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2660.981397] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2661.022313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2661.057382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2661.066854] Lustre: Skipped 2 previous similar messages [ 2663.074290] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2663.138102] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2663.138773] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2663.153448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2669.205440] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2670.216723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2670.233052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2670.377705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2677.098324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2690.180376] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2696.163910] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.165745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2696.181295] Lustre: Skipped 2 previous similar messages [ 2696.200424] Lustre: Skipped 2 previous similar messages [ 2701.642567] Lustre: server umount lustre-MDT0001 complete [ 2702.607887] LustreError: 103617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.625473] LustreError: 103617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2712.301388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.439655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2712.452816] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2712.511886] Lustre: 103612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2713.613624] LustreError: 107466:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2713.626428] LustreError: 107466:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2713.648387] LustreError: 103615:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2713.817020] LustreError: 107483:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.137@tcp: cfg command failed: rc = -17 [ 2713.832725] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2713.832725] [ 2718.355284] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2718.698880] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2727.814043] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:29:25 (1787034565) [ 2745.710403] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2763.138821] Lustre: 109558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2785.220431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2788.284555] Lustre: 110693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2871.777974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2871.790395] LustreError: Skipped 1 previous similar message [ 2871.796403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.822652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.825669] Lustre: Skipped 4 previous similar messages [ 2875.473573] Lustre: server umount lustre-MDT0000 complete [ 2877.416464] LustreError: 103617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.451118] LustreError: 103617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2879.699031] LustreError: 103604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787034718 with bad export cookie 2458608171122704371 [ 2879.701890] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2879.704098] LustreError: 103604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2880.033242] Lustre: server umount lustre-MDT0001 complete [ 2893.669429] Lustre: server umount lustre-OST0000 complete [ 2906.616542] Lustre: server umount lustre-OST0001 complete [ 2911.692835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2920.819975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2929.223627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2941.614263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2960.878843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2971.425177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.444601] Lustre: lustre-MDT0000: reset Object Index mappings [ 2971.928560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.935761] Lustre: Skipped 1 previous similar message [ 2975.937453] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2984.173023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.189133] Lustre: lustre-MDT0001: reset Object Index mappings [ 2989.649018] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2992.746287] Lustre: 115790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2999.669988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.000681] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3009.511123] LustreError: 116145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.533139] LustreError: 116145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3009.552648] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3015.421355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3018.191575] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3018.195413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3018.312125] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3022.746262] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3030.897289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3035.898778] Lustre: 117668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3081.610840] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:35:19 (1787034919) [ 3110.387613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.392410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.404906] Lustre: Skipped 5 previous similar messages [ 3110.418782] Lustre: Skipped 7 previous similar messages [ 3114.557375] Lustre: server umount lustre-MDT0000 complete [ 3117.725255] LustreError: 117670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787034956 with bad export cookie 2458608171122751425 [ 3117.731902] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3117.741746] LustreError: 117670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3132.384554] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3132.616904] Lustre: server umount lustre-MDT0001 complete [ 3150.304566] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3150.408576] Lustre: server umount lustre-OST0000 complete [ 3159.988922] Lustre: server umount lustre-OST0001 complete [ 3165.360689] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3173.420649] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3217.121487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.351266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3217.371221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3217.437814] Lustre: lustre-MDT0000: new disk, initializing [ 3217.527608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.536929] Lustre: Skipped 3 previous similar messages [ 3217.571449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3222.088966] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3229.116168] Lustre: server umount lustre-MDT0000 complete [ 3237.873297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.148690] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3238.186701] Lustre: 121943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3238.222238] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3242.828257] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3247.443974] Lustre: server umount lustre-MDT0000 complete [ 3255.387211] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:38:13 (1787035093) [ 3257.014670] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3259.346440] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:38:16 (1787035096) [ 3280.074766] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3291.142507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.852961] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3306.080281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.173544] Lustre: 124009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3306.219660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3306.223100] Lustre: Skipped 1 previous similar message [ 3306.288930] Lustre: lustre-MDT0001: new disk, initializing [ 3306.397353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3306.417443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3311.545936] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3316.948761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3325.017091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.304902] Lustre: lustre-OST0000: new disk, initializing [ 3325.309980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3325.318497] Lustre: 125588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3327.403633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3327.410340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3327.560311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3332.440845] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3344.705723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3344.950276] Lustre: lustre-OST0001: new disk, initializing [ 3344.964532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3344.972755] Lustre: 126458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3346.523824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3346.545031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3346.716439] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3352.490449] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3363.411335] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3367.074712] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3747.227735] Lustre: 132876:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3747.235760] Lustre: 132876:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3747.242332] Lustre: 132876:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3747.252316] Lustre: 132876:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3747.259920] Lustre: 132876:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3747.266131] Lustre: 132876:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3996.335244] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:50:33 (1787035833) [ 4012.512809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.534253] Lustre: Skipped 4 previous similar messages [ 4012.545524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4012.564914] Lustre: Skipped 11 previous similar messages [ 4017.634495] LustreError: 127289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.658572] LustreError: 127289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 4017.718850] Lustre: server umount lustre-MDT0000 complete [ 4021.393270] LustreError: 124760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787035860 with bad export cookie 2458608171122818135 [ 4021.395821] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4021.415683] LustreError: 124760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4022.753540] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.772150] Lustre: Skipped 2 previous similar messages [ 4022.780649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4022.801032] Lustre: Skipped 1 previous similar message [ 4027.908326] Lustre: server umount lustre-MDT0001 complete [ 4038.399283] Lustre: server umount lustre-OST0000 complete [ 4049.067687] Lustre: server umount lustre-OST0001 complete [ 4056.947362] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4067.924751] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4108.692807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.956326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4108.986400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4109.078886] Lustre: lustre-MDT0000: new disk, initializing [ 4109.175021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.178704] Lustre: Skipped 5 previous similar messages [ 4109.194178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4114.033774] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4124.978522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.127798] Lustre: 135923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4125.188291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4125.191882] Lustre: Skipped 1 previous similar message [ 4125.253860] Lustre: lustre-MDT0001: new disk, initializing [ 4125.395225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4125.403291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4130.122778] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4139.501820] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4144.987642] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4152.649930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.025627] Lustre: lustre-OST0000: new disk, initializing [ 4153.043300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4153.056814] Lustre: 137890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4153.236915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.240036] Lustre: Skipped 1 previous similar message [ 4154.494940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4154.514155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4154.731364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4160.793736] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4170.666996] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4180.255892] Lustre: lustre-MDD0000: changelog on [ 4182.862554] Lustre: lustre-MDD0001: changelog on [ 5905.378869] Lustre: lustre-MDD0001: changelog off [ 5907.227835] Lustre: 135929:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5908.000176] Lustre: lustre-MDD0000: changelog off [ 5916.129082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.130237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.137197] LustreError: Skipped 1 previous similar message [ 5916.141722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5916.160020] Lustre: Skipped 2 previous similar messages [ 5916.181564] Lustre: Skipped 3 previous similar messages [ 5920.611640] Lustre: server umount lustre-OST0000 complete [ 5926.881081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5926.884705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5926.907278] Lustre: Skipped 1 previous similar message [ 5926.916067] Lustre: Skipped 1 previous similar message [ 5932.002833] LustreError: 135929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.022084] LustreError: 135929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5932.761843] Lustre: server umount lustre-MDT0000 complete [ 5937.128993] LustreError: 135934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.168990] LustreError: 135934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5937.205790] LustreError: 135914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787037775 with bad export cookie 2458608171123593287 [ 5937.207568] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5937.220625] LustreError: 135914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5937.549672] Lustre: server umount lustre-MDT0001 complete [ 5950.246504] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5953.009209] Key type lgssc unregistered [ 5953.375979] LNet: 147014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5953.386247] LNetError: 147014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5953.404408] LNet: Removed LNI 192.168.201.137@tcp [ 5954.229150] Key type .llcrypt unregistered [ 5954.231767] Key type ._llcrypt unregistered [ 5980.068958] Key type ._llcrypt registered [ 5980.071166] Key type .llcrypt registered [ 5980.165300] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5997.177189] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5998.443262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5998.547456] alg: No test for adler32 (adler32-zlib) [ 5999.771364] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 6000.152543] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6001.903181] Key type lgssc registered [ 6003.154058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6044.705763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6044.737183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.069604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6046.102499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6046.200977] Lustre: lustre-MDT0000: new disk, initializing [ 6046.295552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.315427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6050.660547] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6062.065290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.256812] Lustre: 150220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6062.330291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6062.335186] Lustre: Skipped 1 previous similar message [ 6062.425304] Lustre: lustre-MDT0001: new disk, initializing [ 6062.525982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6062.570747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6062.597097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6069.412669] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6082.346933] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6088.622462] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6096.617057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.943082] Lustre: lustre-OST0000: new disk, initializing [ 6096.947412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6096.952409] Lustre: 152189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6097.040715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.594509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6102.610507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6102.752353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6103.479397] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6113.722633] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6120.693269] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_module obdecho/obdecho [ 6122.587308] LustreError: 153193:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6122.593273] LustreError: 153193:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6124.450624] LustreError: 153289:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6138.852849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.862695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6138.870856] Lustre: Skipped 2 previous similar messages [ 6138.884163] Lustre: Skipped 2 previous similar messages [ 6142.810359] Lustre: server umount lustre-MDT0000 complete [ 6146.320055] LustreError: 152188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787037984 with bad export cookie 10175550416431104314 [ 6146.329580] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.330889] LustreError: 152188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6146.783697] Lustre: server umount lustre-MDT0001 complete [ 6161.559193] Lustre: server umount lustre-OST0000 complete [ 6189.279830] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6200.507334] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6239.006221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.269415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6239.297423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6239.385129] Lustre: lustre-MDT0000: new disk, initializing [ 6239.437876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6239.454681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6243.965645] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6255.590209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.842486] Lustre: 157009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6255.944770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6255.950416] Lustre: Skipped 1 previous similar message [ 6256.124067] Lustre: lustre-MDT0001: new disk, initializing [ 6256.287840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6256.337430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6256.357038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6261.549617] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6276.108954] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6284.444832] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6294.198922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.869564] Lustre: lustre-OST0000: new disk, initializing [ 6294.887668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6294.901188] Lustre: 158977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6295.108190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.797113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6300.811466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6300.896818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6302.258761] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6311.656607] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6316.821827] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6317.070362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6322.502897] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6322.805657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6326.760266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6326.761780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.778335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6326.804904] Lustre: Skipped 1 previous similar message [ 6331.875337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6331.886683] Lustre: Skipped 2 previous similar messages [ 6336.997482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.021730] Lustre: Skipped 1 previous similar message [ 6339.039696] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6339.287570] Lustre: server umount lustre-OST0000 complete [ 6343.662180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6343.675125] LustreError: Skipped 1 previous similar message [ 6343.681253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.699213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6347.250487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6350.237923] Lustre: server umount lustre-MDT0000 complete [ 6352.355764] LustreError: 157019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.381479] LustreError: 157019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6355.112512] LustreError: 157001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787038193 with bad export cookie 10175550416431105273 [ 6355.120571] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.134434] LustreError: 157001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6355.595834] Lustre: server umount lustre-MDT0001 complete [ 6359.516549] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:29:57 (1787038197) [ 6361.404081] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6363.364226] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:30:00 (1787038200) [ 6371.363035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.963490] LustreError: 161393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.124254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.447743] LustreError: 161394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.475612] LustreError: 161394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6377.607768] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6381.538671] LustreError: 161393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.670197] LustreError: 161394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.499572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.764432] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6401.494118] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6407.201855] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6415.762400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.123038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6416.130856] Lustre: Skipped 1 previous similar message [ 6423.050603] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6431.122027] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6442.315723] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 6442.334447] Lustre: Skipped 2 previous similar messages [ 6442.986464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.024853] Lustre: Skipped 1 previous similar message [ 6448.657660] Lustre: server umount lustre-MDT0000 complete [ 6452.493545] LustreError: 161388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.722824] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787038291 with bad export cookie 10175550416431106155 [ 6452.732978] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6452.733440] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6453.218460] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.233784] Lustre: Skipped 1 previous similar message [ 6459.319160] Lustre: server umount lustre-MDT0001 complete [ 6468.293538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6484.724899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.081831] LustreError: 165237:0:(llog.c:1655:llog_backup()) MGC192.168.201.137@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6485.085946] Lustre: 165237:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.137@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6487.374313] LustreError: 165242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.398825] LustreError: 165242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6522.850990] LustreError: 165260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.882896] LustreError: 165260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 6545.887326] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6545.899435] Lustre: 165237:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6586.861776] LustreError: 165260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.870883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6586.889912] LustreError: 165260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 6591.989578] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6592.783790] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6602.392560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.485892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6622.691771] LustreError: 148441:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b1622779 x1873844988157568/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6628.116194] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6628.362125] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6635.772479] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6637.414456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6644.260288] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.541117] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6669.295358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6669.299467] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.300818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6669.300825] Lustre: Skipped 10 previous similar messages [ 6669.331454] Lustre: Skipped 2 previous similar messages [ 6670.302817] Lustre: server umount lustre-MDT0000 complete [ 6674.620633] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787038513 with bad export cookie 10175550416431112616 [ 6674.650220] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6675.134644] Lustre: server umount lustre-MDT0001 complete [ 6690.324628] Lustre: server umount lustre-OST0000 complete [ 6716.658111] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6730.179828] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6775.302556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6775.766245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6775.854929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6776.015873] Lustre: lustre-MDT0000: new disk, initializing [ 6776.157109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.168542] Lustre: Skipped 1 previous similar message [ 6776.217210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6780.526742] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6791.389348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.599679] Lustre: 170348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6791.968565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6796.677603] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6807.125386] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6813.041884] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6821.139096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6821.420398] Lustre: lustre-OST0000: new disk, initializing [ 6821.423166] Lustre: Skipped 1 previous similar message [ 6821.430367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6821.437538] Lustre: Skipped 2 previous similar messages [ 6821.440977] Lustre: 172313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6828.167864] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6831.690575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6831.698498] Lustre: Skipped 1 previous similar message [ 6831.702347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6831.873545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6838.537402] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6843.781422] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6844.041769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6849.061537] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6849.282132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6852.584277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6852.599258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.616539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6852.620427] Lustre: Skipped 2 previous similar messages [ 6865.375139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6865.581438] Lustre: server umount lustre-OST0000 complete [ 6876.004343] Lustre: server umount lustre-MDT0000 complete [ 6878.190564] LustreError: 170359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.203359] LustreError: 170359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 35 previous similar messages [ 6879.527972] LustreError: 170339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787038718 with bad export cookie 10175550416431129437 [ 6879.534393] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.548737] LustreError: 170339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6880.054274] Lustre: server umount lustre-MDT0001 complete [ 6888.471712] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:38:45 (1787038725) [ 6895.503583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.333825] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6909.859196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.231583] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6923.681674] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6929.759172] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6937.777134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.598799] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6953.355483] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6966.694919] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:40:04 (1787038804) [ 6972.385529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.390685] LustreError: Skipped 1 previous similar message [ 6972.393616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.407422] Lustre: Skipped 3 previous similar messages [ 6972.415372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.422476] Lustre: Skipped 7 previous similar messages [ 6977.901284] Lustre: server umount lustre-OST0000 complete [ 6988.777658] Lustre: server umount lustre-MDT0000 complete [ 6992.918043] LustreError: 177707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787038831 with bad export cookie 10175550416431130319 [ 6992.922061] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6992.933799] LustreError: 177707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6993.373176] Lustre: server umount lustre-MDT0001 complete [ 7005.517032] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7008.392586] Key type lgssc unregistered [ 7008.635501] LNet: 178460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7008.641573] LNetError: 178460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7008.664935] LNet: Removed LNI 192.168.201.137@tcp [ 7009.637101] Key type .llcrypt unregistered [ 7009.641860] Key type ._llcrypt unregistered [ 7028.649479] Key type ._llcrypt registered [ 7028.651631] Key type .llcrypt registered [ 7028.737402] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7030.222532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7030.322917] alg: No test for adler32 (adler32-zlib) [ 7031.373215] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 7031.540049] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7033.207347] Key type lgssc registered [ 7034.405783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7044.097449] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7044.119217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7059.679501] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7064.799333] LustreError: 179704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 7069.920573] LustreError: 179704:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7069.960096] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7069.977500] LustreError: MGC192.168.201.137@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7069.989244] LustreError: 179704:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7069.999867] LustreError: 179704:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7070.010765] LustreError: 179704:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7070.089488] Lustre: server umount lustre-OST0000 complete [ 7070.096965] LustreError: 179704:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7079.395655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.995042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.528844] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7091.178362] LustreError: 180204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7091.195241] LustreError: 180204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7094.427799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.865701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7099.435403] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7106.913879] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7112.530570] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7119.374541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.548741] LustreError: 182021:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7119.561288] Lustre: 182021:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.137@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7119.696514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.168397] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7133.334538] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.717524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7135.731152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.747747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.349376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.358133] Lustre: Skipped 5 previous similar messages [ 7141.853115] Lustre: server umount lustre-MDT0000 complete [ 7146.002644] LustreError: 180184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787038984 with bad export cookie 8528142645194565734 [ 7146.020316] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.470357] LustreError: 180201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.489439] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.532821] Lustre: Skipped 2 previous similar messages [ 7146.916906] Lustre: server umount lustre-MDT0001 complete [ 7153.794572] Lustre: server umount lustre-OST0000 complete [ 7164.457132] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7176.695797] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7215.949274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.186956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7216.235461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7216.362491] Lustre: lustre-MDT0000: new disk, initializing [ 7216.477956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.508447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7220.947674] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7230.274187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.386163] Lustre: 185549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7230.443675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7230.446112] Lustre: Skipped 1 previous similar message [ 7230.492183] Lustre: lustre-MDT0001: new disk, initializing [ 7230.561359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7230.604343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7230.636012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7234.128257] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7242.862869] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7247.778837] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7253.597519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7253.875697] Lustre: lustre-OST0000: new disk, initializing [ 7253.880907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7253.886209] Lustre: 187517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7253.966519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.089640] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7262.278213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7262.292158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7262.365724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7268.659862] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.027796] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7273.177970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7277.312533] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7277.514856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7283.168666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7283.169674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.185630] LustreError: Skipped 1 previous similar message [ 7283.187165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7283.200833] Lustre: Skipped 1 previous similar message [ 7283.212859] Lustre: Skipped 1 previous similar message [ 7285.796553] Lustre: server umount lustre-OST0000 complete [ 7291.882175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7291.894404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.907538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7295.740232] Lustre: server umount lustre-MDT0000 complete [ 7298.528308] LustreError: 185560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7298.550106] LustreError: 185560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7299.334432] LustreError: 185540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787039137 with bad export cookie 8528142645194566455 [ 7299.341347] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.344059] LustreError: 185540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7299.644486] Lustre: server umount lustre-MDT0001 complete [ 7307.606427] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:45:45 (1787039145) [ 7323.033262] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7325.320475] Key type lgssc unregistered [ 7325.553439] LNet: 190461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7325.559950] LNetError: 190461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7325.587456] LNet: Removed LNI 192.168.201.137@tcp [ 7326.272427] Key type .llcrypt unregistered [ 7326.274448] Key type ._llcrypt unregistered [ 7342.374069] Key type ._llcrypt registered [ 7342.378380] Key type .llcrypt registered [ 7342.494780] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7343.615435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7343.664897] alg: No test for adler32 (adler32-zlib) [ 7344.703268] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 7344.921899] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7346.559177] Key type lgssc registered [ 7347.636146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7354.161590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7354.185709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.599504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.331843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7366.115381] LustreError: 191678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7366.138699] LustreError: 191678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7366.858723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.122846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7370.662481] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7376.950601] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7382.481262] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7390.348969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7390.588267] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7390.592243] LustreError: 193542:0:(llog.c:1696:llog_backup()) MGC192.168.201.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7390.600258] Lustre: 193542:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.137@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7390.729245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7396.297608] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7402.829653] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7406.563488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.566472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7406.584683] Lustre: Skipped 1 previous similar message [ 7406.595023] Lustre: Skipped 2 previous similar messages [ 7410.880271] Lustre: server umount lustre-MDT0000 complete [ 7411.682173] LustreError: 191673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7411.696716] LustreError: 191673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7414.217318] LustreError: 191660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787039252 with bad export cookie 11052406583134152715 [ 7414.222947] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7414.227599] LustreError: 191660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7414.490628] Lustre: server umount lustre-MDT0001 complete [ 7428.217587] Lustre: server umount lustre-OST0000 complete [ 7436.633447] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7445.371973] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7474.076393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.273587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7474.295040] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7474.377579] Lustre: lustre-MDT0000: new disk, initializing [ 7474.446812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.459975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7477.887650] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7486.299210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.398420] Lustre: 197065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7486.436525] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7486.438591] Lustre: Skipped 1 previous similar message [ 7486.440746] LustreError: 197846:0:(llog.c:1696:llog_backup()) MGC192.168.201.137@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7486.444994] LustreError: 197846:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7486.448086] Lustre: 197846:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.137@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7486.462863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7486.465796] Lustre: Skipped 1 previous similar message [ 7486.514368] Lustre: lustre-MDT0001: new disk, initializing [ 7486.584950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7486.611567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7486.623264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7490.285127] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7498.637061] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7502.924441] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7509.109495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7509.296978] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7509.299950] Lustre: Skipped 1 previous similar message [ 7509.301914] LustreError: 199006:0:(llog.c:1696:llog_backup()) MGC192.168.201.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7509.307369] LustreError: 199006:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7509.321760] Lustre: 199006:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.137@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7509.328684] Lustre: 199006:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7509.352034] Lustre: lustre-OST0000: new disk, initializing [ 7509.356915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7509.363908] Lustre: 199030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7509.453675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7514.487850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7517.780459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7517.803903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7517.892169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7522.296533] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7526.341280] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7526.533540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7529.994040] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7530.168306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7533.537076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7533.538605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.540719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.541463] LustreError: Skipped 1 previous similar message [ 7533.564864] Lustre: Skipped 2 previous similar messages [ 7537.688711] Lustre: server umount lustre-OST0000 complete [ 7543.271172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7543.278450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.289597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.295296] Lustre: Skipped 1 previous similar message [ 7547.327229] Lustre: server umount lustre-MDT0000 complete [ 7548.896057] LustreError: 197076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.912754] LustreError: 197076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7550.084476] LustreError: 199029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787039388 with bad export cookie 11052406583134153436 [ 7550.085923] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.104191] LustreError: 199029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7550.444474] Lustre: server umount lustre-MDT0001 complete [ 7557.085233] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:49:55 (1787039395) [ 7568.444406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7575.350901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.613328] LustreError: 202155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7575.652274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.622358] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7581.156490] LustreError: 202156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.129264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7587.059373] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7588.988289] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7593.601265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7598.114436] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7604.109277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7604.278711] Lustre: lustre-OST0001: new disk, initializing [ 7604.282208] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7604.288747] Lustre: 204460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7607.523725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7607.537124] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7607.624215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7609.487389] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7619.154222] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7623.228941] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7624.686422] Lustre: DEBUG MARKER: ADD OST3 [ 7628.038840] loop: module loaded [ 7628.102651] loop0: detected capacity change from 0 to 204800000 [ 7628.349924] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7628.355690] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7628.361358] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7632.765392] loop0: detected capacity change from 0 to 204800000 [ 7634.613494] Lustre: DEBUG MARKER: STOP OST3 [ 7637.688602] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7637.864758] Lustre: lustre-OST0003: new disk, initializing [ 7637.869130] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7637.877449] Lustre: 206389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7638.002917] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7638.008893] Lustre: Skipped 3 previous similar messages [ 7639.090624] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7639.180219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7639.185686] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7639.220753] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7639.455359] Lustre: Failing over lustre-OST0003 [ 7639.550667] Lustre: server umount lustre-OST0003 complete [ 7639.723950] LustreError: 203639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7641.119177] LustreError: 206536:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7643.378352] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7644.645079] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7644.651281] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.658048] Lustre: Skipped 1 previous similar message [ 7649.925786] Lustre: DEBUG MARKER: START OST3 again [ 7654.081839] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7654.274034] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7655.186811] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7655.193414] Lustre: lustre-OST0003: Denying connection for new client 81d7a19c-8ed8-42e2-bc5c-9b9c139096d5 (at 192.168.201.37@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 7656.254207] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7656.254315] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7658.132935] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7674.956016] Lustre: Failing over lustre-OST0003 [ 7675.044512] Lustre: server umount lustre-OST0003 complete [ 7675.693659] LustreError: 203641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.704473] LustreError: 203641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7676.900731] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7676.901415] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.911913] LustreError: Skipped 1 previous similar message [ 7676.936829] Lustre: Skipped 2 previous similar messages [ 7690.217146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7690.234931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7690.243061] Lustre: Skipped 1 previous similar message [ 7695.331052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.339361] Lustre: Skipped 6 previous similar messages [ 7696.541759] Lustre: server umount lustre-MDT0000 complete [ 7698.977687] LustreError: 205228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787039537 with bad export cookie 11052406583134154318 [ 7699.014973] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7704.483738] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7704.486666] Lustre: Skipped 2 previous similar messages [ 7705.277706] Lustre: server umount lustre-MDT0001 complete [ 7708.009366] Lustre: server umount lustre-OST0000 complete [ 7710.997521] Lustre: server umount lustre-OST0001 complete [ 7715.216276] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7720.470165] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7750.440289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.682839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7750.747640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7750.879619] Lustre: lustre-MDT0000: new disk, initializing [ 7751.001837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7751.004757] Lustre: Skipped 1 previous similar message [ 7751.039324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7756.847239] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7774.601757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.967667] Lustre: 211912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7775.244339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7775.267349] Lustre: Skipped 1 previous similar message [ 7775.636727] Lustre: lustre-MDT0001: new disk, initializing [ 7776.355205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7776.401624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7784.048263] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7800.754398] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7809.173351] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7816.697544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.028582] Lustre: lustre-OST0000: new disk, initializing [ 7817.032690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7817.041484] Lustre: 213884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7818.891115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7818.904744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7819.047446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7823.846922] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7832.766495] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7837.840520] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7838.075874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7841.997227] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7842.238410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7844.837078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7844.840917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.848212] LustreError: Skipped 1 previous similar message [ 7844.850691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7844.860357] Lustre: Skipped 6 previous similar messages [ 7844.879359] Lustre: Skipped 2 previous similar messages [ 7850.416365] Lustre: server umount lustre-OST0000 complete [ 7868.388237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7868.728603] Lustre: server umount lustre-MDT0000 complete [ 7870.432620] LustreError: 211923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.450547] LustreError: 211923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 7872.804754] LustreError: 211903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787039711 with bad export cookie 11052406583134156600 [ 7872.819145] LustreError: 211903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7872.822563] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7873.242686] Lustre: server umount lustre-MDT0001 complete [ 7876.976368] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:55:14 (1787039714) [ 7894.000590] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7904.241927] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7943.132245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.496281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7943.536247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7943.659929] Lustre: lustre-MDT0000: new disk, initializing [ 7943.760827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7943.764607] Lustre: Skipped 2 previous similar messages [ 7943.779865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7948.587311] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7960.682679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.756316] Lustre: 218495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7961.025278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7965.686287] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7974.136054] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7979.588972] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7987.062685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.345200] Lustre: 220464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7989.433138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7994.398412] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8004.235342] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8009.179755] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8009.535704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8014.784391] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8015.041942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8019.938964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8019.954491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.983680] Lustre: Skipped 2 previous similar messages [ 8019.996725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8020.014313] Lustre: Skipped 8 previous similar messages [ 8023.727600] Lustre: server umount lustre-OST0000 complete [ 8035.124942] Lustre: server umount lustre-MDT0000 complete [ 8035.813422] LustreError: 219362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.761276] LustreError: 218486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787039878 with bad export cookie 11052406583134157482 [ 8039.767244] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.772679] LustreError: 218486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8040.160324] Lustre: server umount lustre-MDT0001 complete [ 8050.279997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.012728] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8064.573806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.224969] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8078.638877] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8084.462345] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8092.135916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.578502] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8105.992920] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8119.782763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8119.789778] LustreError: Skipped 1 previous similar message [ 8119.799618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.813401] Lustre: Skipped 3 previous similar messages [ 8119.822874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8119.825386] Lustre: Skipped 4 previous similar messages [ 8122.795274] Lustre: server umount lustre-OST0000 complete [ 8133.942169] Lustre: server umount lustre-MDT0000 complete [ 8138.275676] LustreError: 222661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787039976 with bad export cookie 11052406583134158371 [ 8138.302606] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8138.569726] Lustre: server umount lustre-MDT0001 complete [ 8245.835586] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8248.900802] Key type lgssc unregistered [ 8249.208532] LNet: 226354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8249.214979] LNetError: 226354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8249.231114] LNet: Removed LNI 192.168.201.137@tcp [ 8250.090204] Key type .llcrypt unregistered [ 8250.091675] Key type ._llcrypt unregistered [ 8276.765363] Key type ._llcrypt registered [ 8276.767512] Key type .llcrypt registered [ 8276.884161] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8294.001443] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8295.336199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8295.350224] alg: No test for adler32 (adler32-zlib) [ 8296.673135] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 8296.902747] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8298.568084] Key type lgssc registered [ 8299.701634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8336.664510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8336.689377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.047522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8338.107388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8338.189406] Lustre: lustre-MDT0000: new disk, initializing [ 8338.308796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.340588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8342.431741] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8353.577154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.847848] Lustre: 229722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8353.995831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8354.000388] Lustre: Skipped 1 previous similar message [ 8354.119698] Lustre: lustre-MDT0001: new disk, initializing [ 8354.223588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8354.270502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8354.285889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8359.220622] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8370.162592] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8375.379993] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8382.981158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8383.283956] Lustre: lustre-OST0000: new disk, initializing [ 8383.288328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8383.298066] Lustre: 231690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8383.380471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.100738] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8390.745626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8390.755122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8390.909265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8400.016921] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8404.427412] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8404.667930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8408.593947] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8408.786256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8411.622469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8411.625953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.635387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.654785] Lustre: Skipped 1 previous similar message [ 8416.733933] Lustre: server umount lustre-OST0000 complete [ 8420.832156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8420.849721] LustreError: Skipped 1 previous similar message [ 8420.852424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.874088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8420.881434] Lustre: Skipped 1 previous similar message [ 8421.857159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.874740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8421.886151] Lustre: Skipped 1 previous similar message [ 8426.977357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8426.986283] Lustre: Skipped 1 previous similar message [ 8432.098592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8432.108508] Lustre: Skipped 1 previous similar message [ 8435.168560] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8435.375224] Lustre: server umount lustre-MDT0000 complete [ 8438.786426] LustreError: 229713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787040277 with bad export cookie 11727008716512618287 [ 8438.795762] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8438.802848] LustreError: 229713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8439.036907] Lustre: server umount lustre-MDT0001 complete [ 8458.462860] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8471.344910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8471.886552] LustreError: 234624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8471.895732] LustreError: 234624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8471.948514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8476.952921] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8477.154624] LustreError: 234625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8482.275382] LustreError: 234624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8487.413984] LustreError: 234625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.469869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.922492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8494.394407] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8497.370962] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8505.276660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8505.754577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8512.476229] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8521.897172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.094753] Lustre: lustre-OST0001: new disk, initializing [ 8522.099056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8522.104678] Lustre: 236930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8522.211584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8528.708617] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8529.548700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8529.565234] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8529.702096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8540.601296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8549.947946] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8562.419202] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 04:06:39 (1787040399) [ 8565.731569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8565.742820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.750348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8570.982437] LustreError: 238060:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffffa0c607e5c700) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8571.158622] Lustre: server umount lustre-MDT0000 complete [ 8574.743846] LustreError: 236929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787040413 with bad export cookie 11727008716512619169 [ 8574.751765] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8574.767694] LustreError: 236929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8574.994215] Lustre: server umount lustre-MDT0001 complete [ 8589.027312] Lustre: server umount lustre-OST0000 complete [ 8592.352160] Lustre: 227946:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787040414/real 1787040414] req@ffffa0c62c7e8a80 x1873847396710656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787040430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8592.389037] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.405168] Lustre: Skipped 3 previous similar messages [ 8593.104652] Lustre: server umount lustre-OST0001 complete [ 8600.318904] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8611.736026] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8648.403726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.624457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8648.690221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8648.825402] Lustre: lustre-MDT0000: new disk, initializing [ 8648.908476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8648.934213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8653.036855] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8663.179911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8663.336252] Lustre: 240876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8663.392849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8663.397084] Lustre: Skipped 1 previous similar message [ 8663.453394] Lustre: lustre-MDT0001: new disk, initializing [ 8663.595397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8663.607177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8667.478570] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8676.533420] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8682.211964] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8689.082535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8689.369747] Lustre: lustre-OST0000: new disk, initializing [ 8689.373895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8689.379880] Lustre: 242845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8690.723501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8690.728917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8690.799801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8695.496553] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8705.157769] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8710.311848] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8710.540381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8715.447383] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8715.725237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8721.376239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8721.380519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8721.388321] LustreError: Skipped 1 previous similar message [ 8721.390537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8721.411100] Lustre: Skipped 1 previous similar message [ 8721.435632] Lustre: Skipped 8 previous similar messages [ 8723.972438] Lustre: server umount lustre-OST0000 complete [ 8730.081842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8742.367268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8742.612980] Lustre: server umount lustre-MDT0000 complete [ 8746.377969] LustreError: 240869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787040585 with bad export cookie 11727008716512621297 [ 8746.381329] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8746.384434] LustreError: 240869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8746.659209] Lustre: server umount lustre-MDT0001 complete [ 8754.354394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.621827] LustreError: 245064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.672452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8754.674728] Lustre: Skipped 2 previous similar messages [ 8759.643265] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8764.899034] LustreError: 245065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.912825] LustreError: 245065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8766.696348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.034218] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8776.896325] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8781.283338] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8786.868418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.727889] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8799.171411] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8946.151097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.166430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8946.179146] Lustre: Skipped 2 previous similar messages [ 8946.183496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.192864] Lustre: Skipped 7 previous similar messages [ 8950.476858] Lustre: server umount lustre-OST0000 complete [ 8968.159189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8968.334716] Lustre: server umount lustre-MDT0000 complete [ 8968.672432] LustreError: 245065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8968.701266] LustreError: 245065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8971.916276] LustreError: 247885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787040810 with bad export cookie 11727008716512622186 [ 8971.927261] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8971.934455] LustreError: 247885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8972.292901] Lustre: server umount lustre-MDT0001 complete [ 9068.468354] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9070.722607] Key type lgssc unregistered [ 9070.983695] LNet: 248674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9070.988816] LNetError: 248674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9071.001542] LNet: Removed LNI 192.168.201.137@tcp [ 9071.733169] Key type .llcrypt unregistered [ 9071.741168] Key type ._llcrypt unregistered [ 9094.819701] Key type ._llcrypt registered [ 9094.821156] Key type .llcrypt registered [ 9094.923553] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 9109.004632] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9109.687969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9109.702816] alg: No test for adler32 (adler32-zlib) [ 9110.668232] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [ 9110.857384] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9112.490963] Key type lgssc registered [ 9113.268747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9155.341930] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9165.033817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9165.079663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.360660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9166.396708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9166.472980] Lustre: lustre-MDT0000: new disk, initializing [ 9166.565449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9166.608170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9171.373872] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9181.027902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9181.134875] Lustre: 252745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9181.187951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9181.193900] Lustre: Skipped 1 previous similar message [ 9181.271309] Lustre: lustre-MDT0001: new disk, initializing [ 9181.348249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9181.386985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9181.402701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9185.995862] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9191.371420] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9197.958636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9198.193636] Lustre: lustre-OST0000: new disk, initializing [ 9198.198314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9198.204315] Lustre: 254373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9198.309076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9203.788811] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9206.890933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9206.906066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9207.038233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9212.700283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9212.849905] Lustre: lustre-OST0001: new disk, initializing [ 9212.853220] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9212.860387] Lustre: 255241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9212.940774] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9215.981164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9216.002494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9216.185735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9219.913338] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9229.500406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9235.458578] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9244.409605] Lustre: server umount lustre-MDT0000 complete [ 9246.700223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9246.727982] Lustre: Skipped 3 previous similar messages [ 9247.373599] LustreError: 252735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787041086 with bad export cookie 4770545279298147342 [ 9247.376693] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9247.383418] LustreError: 252735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9249.551849] Lustre: server umount lustre-MDT0001 complete [ 9263.093646] Lustre: server umount lustre-OST0000 complete [ 9276.777434] Lustre: server umount lustre-OST0001 complete [ 9288.790542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9297.117143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9305.182631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9314.074107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9332.884564] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9342.438150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.479162] Lustre: lustre-MDT0000: reset Object Index mappings [ 9343.002373] LustreError: 260252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.020451] LustreError: 260252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9343.097338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.583634] LustreError: 260253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.014187] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9352.678500] LustreError: 260252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9357.217664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.244261] Lustre: lustre-MDT0001: reset Object Index mappings [ 9357.658035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9362.508427] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9365.565324] Lustre: 261395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9371.445310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9377.089367] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9383.160586] LustreError: 261749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.486256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9385.704537] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9385.715222] Lustre: Skipped 1 previous similar message [ 9387.806221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9387.810041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9391.912573] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9399.193187] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9402.201651] Lustre: 263266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9408.497540] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9408.501635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9408.513130] Lustre: Skipped 3 previous similar messages [ 9408.525842] Lustre: Skipped 3 previous similar messages [ 9412.052210] Lustre: server umount lustre-MDT0000 complete [ 9413.612857] LustreError: 260985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9413.628143] LustreError: 260985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9415.449428] LustreError: 260232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787041254 with bad export cookie 4770545279298154132 [ 9415.463947] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9415.463948] LustreError: 260232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9415.708861] Lustre: server umount lustre-MDT0001 complete [ 9429.377307] Lustre: server umount lustre-OST0000 complete [ 9443.569146] Lustre: server umount lustre-OST0001 complete [ 9459.730391] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:21:37 (1787041297) [ 9474.329102] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 9482.922581] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9513.279481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9513.540491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9513.564414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9513.625831] Lustre: lustre-MDT0000: new disk, initializing [ 9513.692217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9513.715408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9517.584577] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9525.469634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9525.588953] Lustre: 267523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9525.633176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9525.636868] Lustre: Skipped 1 previous similar message [ 9525.686079] Lustre: lustre-MDT0001: new disk, initializing [ 9525.809914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9525.824776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9529.342283] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9537.084687] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9541.540914] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9548.809728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9549.177739] Lustre: lustre-OST0000: new disk, initializing [ 9549.182648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9549.187817] Lustre: 269487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9550.315124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9550.321161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9550.420325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9554.439498] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9561.890722] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9566.094270] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9566.299633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9569.729321] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9569.903677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9575.904427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9575.911848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.916900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9575.932884] Lustre: Skipped 1 previous similar message [ 9575.941293] Lustre: Skipped 1 previous similar message [ 9577.565122] Lustre: server umount lustre-OST0000 complete [ 9581.025358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.026138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9581.033379] Lustre: Skipped 1 previous similar message [ 9586.151387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9586.159544] Lustre: Skipped 2 previous similar messages [ 9587.060545] Lustre: server umount lustre-MDT0000 complete [ 9590.272405] LustreError: 267513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787041428 with bad export cookie 4770545279298156225 [ 9590.275540] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9590.281013] LustreError: 267513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9590.503703] Lustre: server umount lustre-MDT0001 complete [ 9605.999316] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9614.791274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.153305] LustreError: 272420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9615.215085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9615.219230] Lustre: Skipped 2 previous similar messages [ 9618.800686] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9626.490556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.790936] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9633.118053] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9638.439665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9643.439267] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9649.677409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9649.843597] Lustre: lustre-OST0001: new disk, initializing [ 9649.846057] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9649.850300] Lustre: 274726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9650.983868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9651.000303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9651.072846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9655.588315] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9664.027541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9666.806758] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9671.495135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9671.556514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9678.304305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9678.317384] LustreError: Skipped 1 previous similar message [ 9678.329737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.347179] LustreError: 273149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.353894] LustreError: 273149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9678.405939] Lustre: server umount lustre-MDT0000 complete [ 9681.581649] LustreError: 272400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787041520 with bad export cookie 4770545279298157107 [ 9681.583023] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9681.590208] LustreError: 272400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9681.796913] Lustre: server umount lustre-MDT0001 complete [ 9694.704827] Lustre: server umount lustre-OST0000 complete [ 9707.423573] Lustre: server umount lustre-OST0001 complete [ 9720.346786] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9727.169794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.404730] LustreError: 277780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9727.415733] LustreError: 277780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9730.318508] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9735.621641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.153975] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9740.004313] Lustre: 278919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9744.441240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9744.674547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9744.682540] Lustre: Skipped 5 previous similar messages [ 9748.186498] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9753.085557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9754.706751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9759.908489] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9760.249901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9766.471516] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9769.153778] Lustre: 280793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9772.540448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9772.552296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9779.332643] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:26:57 (1787041617) [ 9796.074956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.085977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9796.091826] Lustre: Skipped 5 previous similar messages [ 9796.099586] Lustre: Skipped 1 previous similar message [ 9799.119635] Lustre: server umount lustre-MDT0000 complete [ 9801.191692] LustreError: 277777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.205810] LustreError: 277777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 9801.656188] LustreError: 277761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787041640 with bad export cookie 4770545279298163407 [ 9801.659631] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.670504] LustreError: 277761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9806.308587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9806.317674] Lustre: Skipped 3 previous similar messages [ 9807.958468] Lustre: server umount lustre-MDT0001 complete [ 9825.247147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9825.348775] Lustre: server umount lustre-OST0000 complete [ 9834.208260] Lustre: server umount lustre-OST0001 complete [ 9838.703425] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 9845.859281] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9877.416780] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9885.149971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.323697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9885.342244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9885.390788] Lustre: lustre-MDT0000: new disk, initializing [ 9885.507759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9888.767148] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9897.150049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.255434] Lustre: 285029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9897.503526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9900.833427] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9910.597408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9910.859694] Lustre: 286660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9912.462312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9917.614083] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9926.550520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9926.678619] Lustre: 287533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9928.063622] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9931.530536] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9939.062227] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9941.786480] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9941.789752] Lustre: Skipped 1 previous similar message [ 9952.275564] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:29:50 (1787041790) [ 9958.888865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.899478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9958.911426] Lustre: Skipped 5 previous similar messages [ 9958.914575] Lustre: Skipped 1 previous similar message [ 9958.932077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9960.282588] Lustre: server umount lustre-MDT0000 complete [ 9962.978879] LustreError: 285020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787041801 with bad export cookie 4770545279298171282 [ 9962.983564] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9962.990319] LustreError: 285020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9963.297871] Lustre: server umount lustre-MDT0001 complete [ 9976.414813] Lustre: server umount lustre-OST0000 complete [ 9979.363544] Lustre: 250249:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787041802/real 1787041802] req@ffffa0c607cd3800 x1873848250492416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787041818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9979.809334] Lustre: server umount lustre-OST0001 complete [ 9986.437996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.737184] LustreError: 289782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.753660] LustreError: 289782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9990.097575] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9995.908954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9999.080423] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10004.572227] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10008.525498] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10013.541491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.784886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10013.790114] Lustre: Skipped 7 previous similar messages [10018.582859] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10023.719480] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10045.483222] loop1: detected capacity change from 0 to 204800000 [10045.620031] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10045.632727] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10045.651848] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10049.874211] loop1: detected capacity change from 0 to 204800000 [10053.866720] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.927354] LustreError: 289771:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1818: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10053.936936] LustreError: 289771:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1818 [10053.940897] LustreError: 289771:0:(mgs_handler.c:517:mgs_target_reg()) fs1818-MDT0001: Error checking target index: rc = -13 [10053.951219] LustreError: fs1818-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10053.961487] LustreError: 293517:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10053.968594] LustreError: 293517:0:(tgt_mount.c:2129:server_put_super()) no obd fs1818-MDT0001 [10053.974413] LustreError: 293517:0:(tgt_mount.c:116:server_deregister_mount()) fs1818-MDT0001 not registered [10054.022972] Lustre: server umount fs1818-MDT0001 complete [10054.024823] LustreError: 293517:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10061.025854] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.121956] Lustre: Setting parameter fs1818-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1818-MDT0001 [10061.189042] Lustre: srv-fs1818-MDT0001: No data found on store. Initialize space. [10061.193207] Lustre: Skipped 4 previous similar messages [10061.283167] Lustre: fs1818-MDT0001: new disk, initializing [10061.293800] Lustre: Skipped 3 previous similar messages [10061.403071] Lustre: srv-fs1818-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10061.407049] Lustre: 293984:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1818-MDT0001-mdtlov: can't create new llog: rc = -115 [10061.416336] LustreError: 293984:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1818-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10064.772318] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10070.334910] Lustre: server umount fs1818-MDT0001 complete [10079.666796] loop1: detected capacity change from 0 to 204800000 [10084.095481] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.343524] Lustre: 295552:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1818-MDT0001-mdtlov: can't create new llog: rc = -115 [10084.353271] LustreError: 295552:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1818-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [10088.560957] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10092.306691] Lustre: server umount fs1818-MDT0001 complete [10101.028371] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [10101.039183] Lustre: Skipped 2 previous similar messages [10103.268715] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10103.279326] Lustre: Skipped 3 previous similar messages [10105.606244] Lustre: server umount lustre-MDT0000 complete [10108.387985] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10111.547302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10111.662118] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10115.021638] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10117.102279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10121.119826] loop1: detected capacity change from 0 to 204800000 [10125.021621] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.197357] Lustre: 297996:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1818-MDT0001-mdtlov: can't create new llog: rc = -115 [10125.202380] LustreError: 297996:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1818-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10128.056512] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10131.139994] Lustre: server umount fs1818-MDT0001 complete [10147.809282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10160.095498] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10160.228503] Lustre: server umount lustre-OST0000 complete [10163.169698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10168.288183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.295431] Lustre: Skipped 12 previous similar messages [10169.197135] Lustre: server umount lustre-MDT0000 complete [10172.266046] LustreError: 289762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787042010 with bad export cookie 4770545279298173963 [10172.268382] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10172.279124] LustreError: 289762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10172.518819] Lustre: server umount lustre-MDT0001 complete [10180.349151] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10182.521687] Key type lgssc unregistered [10182.776188] LNet: 300337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10182.779955] LNetError: 300337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10183.852766] LNet: Removed LNI 192.168.201.137@tcp [10184.742377] Key type .llcrypt unregistered [10184.744096] Key type ._llcrypt unregistered [10198.418798] Key type ._llcrypt registered [10198.422970] Key type .llcrypt registered [10198.496974] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10199.339762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10199.362634] alg: No test for adler32 (adler32-zlib) [10200.351192] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [10200.535346] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10202.160671] Key type lgssc registered [10203.011690] Lustre: Echo OBD driver; http://www.lustre.org/ [10208.262629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10208.279133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10209.606126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10213.000475] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10218.955741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.286850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10222.400806] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10227.114157] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10230.709946] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10235.291843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10235.586274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10240.058953] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10243.821171] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10243.844480] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10246.005851] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10249.191563] LustreError: 303417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10254.319636] LustreError: 303417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.683475] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.694146] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10261.786851] LustreError: 303417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.797539] LustreError: 303417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10268.603625] loop1: detected capacity change from 0 to 204800000 [10268.783600] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10268.806074] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10268.829398] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10270.691147] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.699787] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10272.793144] loop1: detected capacity change from 0 to 204800000 [10276.874473] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10277.020819] Lustre: fs7873-OST0000: new disk, initializing [10277.024792] Lustre: srv-fs7873-OST0000: No data found on store. Initialize space. [10277.030505] Lustre: 305342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10277.092674] Lustre: fs7873-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10281.480129] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10287.373211] LustreError: 304003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10287.383816] LustreError: 304003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10287.936310] Lustre: server umount fs7873-OST0000 complete [10296.429718] loop1: detected capacity change from 0 to 204800000 [10299.963628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10300.113423] Lustre: fs7873-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10303.804844] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10309.864601] loop2: detected capacity change from 0 to 204800000 [10310.046809] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10310.054322] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10310.059881] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10313.310957] loop2: detected capacity change from 0 to 204800000 [10317.159712] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10317.239607] LustreError: fs7873-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10317.248533] LustreError: fs7873-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10317.255618] LustreError: 308376:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10317.261523] LustreError: 308376:0:(tgt_mount.c:2129:server_put_super()) no obd fs7873-OST0001 [10317.265742] LustreError: 308376:0:(tgt_mount.c:116:server_deregister_mount()) fs7873-OST0001 not registered [10317.293213] Lustre: server umount fs7873-OST0001 complete [10317.296217] LustreError: 308376:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10321.898064] LustreError: 303418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10321.913302] LustreError: 303418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10323.836559] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.928987] Lustre: fs7873-OST0001: new disk, initializing [10323.930937] Lustre: srv-fs7873-OST0001: No data found on store. Initialize space. [10323.934578] Lustre: 308926:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10323.978896] Lustre: fs7873-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10327.021371] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10328.229887] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10334.524904] Lustre: server umount fs7873-OST0000 complete [10337.256324] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10340.109836] Lustre: server umount fs7873-OST0001 complete [10348.673687] loop1: detected capacity change from 0 to 204800000 [10352.535648] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10352.668485] Lustre: fs7873-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10356.655510] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10361.395864] loop2: detected capacity change from 0 to 204800000 [10365.083422] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10369.721651] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10375.979561] Lustre: server umount fs7873-OST0000 complete [10382.093700] Lustre: server umount fs7873-OST0001 complete [10388.450238] LustreError: 303418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.468559] LustreError: 303418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10392.548573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10392.558682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.568943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10393.569136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.571448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10393.584817] Lustre: Skipped 1 previous similar message [10398.692600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10398.700500] Lustre: Skipped 1 previous similar message [10403.811667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10403.821119] Lustre: Skipped 1 previous similar message [10404.319156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10404.414112] Lustre: server umount lustre-OST0000 complete [10408.931940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.938051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.953024] Lustre: Skipped 1 previous similar message [10408.971673] Lustre: Skipped 1 previous similar message [10412.917536] Lustre: server umount lustre-MDT0000 complete [10415.677017] LustreError: 303441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787042254 with bad export cookie 15896166495603033648 [10415.682739] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10415.685333] LustreError: 303441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10415.945675] Lustre: server umount lustre-MDT0001 complete [10427.760458] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10433.594901] Key type lgssc unregistered [10434.150225] LNet: 314846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10434.556742] LNetError: 314846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10434.645434] LNet: Removed LNI 192.168.201.137@tcp [10435.350170] Key type .llcrypt unregistered [10435.352243] Key type ._llcrypt unregistered [10473.672611] Key type ._llcrypt registered [10473.675602] Key type .llcrypt registered [10473.794053] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [10492.934213] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10494.954544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10495.146377] alg: No test for adler32 (adler32-zlib) [10496.211852] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [10496.533374] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10498.351273] Key type lgssc registered [10500.535415] Lustre: Echo OBD driver; http://www.lustre.org/ [10536.583915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10536.637187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10537.876882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10537.906639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10538.026018] Lustre: lustre-MDT0000: new disk, initializing [10538.168252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10538.186238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10542.083068] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10550.992092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.121835] Lustre: 318708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10551.162152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10551.164953] Lustre: Skipped 1 previous similar message [10551.213443] Lustre: lustre-MDT0001: new disk, initializing [10551.283608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10551.317479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10551.341430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10554.989482] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10563.943972] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10569.626090] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10575.364223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10575.628593] Lustre: lustre-OST0000: new disk, initializing [10575.632927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10575.638262] Lustre: 320675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10575.730172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10581.438689] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10584.635551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10584.644829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10584.730090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10589.386582] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10591.182058] Lustre: Mounted lustre-client [10592.031510] Lustre: Unmounted lustre-client [10600.422060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10600.426392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10600.428688] LustreError: Skipped 1 previous similar message [10600.439425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10601.459270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10601.464664] Lustre: Skipped 2 previous similar messages [10605.220773] Lustre: server umount lustre-OST0000 complete [10611.684428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10611.689576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10611.691143] Lustre: Skipped 2 previous similar messages [10611.696626] Lustre: Skipped 1 previous similar message [10615.005279] Lustre: server umount lustre-MDT0000 complete [10617.792562] LustreError: 318699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787042456 with bad export cookie 11452087623515723599 [10617.794081] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10617.807069] LustreError: 318699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10618.108501] Lustre: server umount lustre-MDT0001 complete [10626.991274] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10629.281615] Key type lgssc unregistered [10653.087702] LNet: 322835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10653.100129] LNetError: 322835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10720.678803] LNet: Removed LNI 192.168.201.137@tcp [10721.899466] Key type .llcrypt unregistered [10721.902685] Key type ._llcrypt unregistered [10738.779601] Key type ._llcrypt registered [10738.782304] Key type .llcrypt registered [10738.972549] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [10753.021989] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10753.827945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10754.103171] alg: No test for adler32 (adler32-zlib) [10755.115489] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [10755.328856] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10756.999241] Key type lgssc registered [10758.012919] Lustre: Echo OBD driver; http://www.lustre.org/ [10783.882345] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:43:41 (1787042621) [10789.837447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10789.862139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.147964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10791.190679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10791.286260] Lustre: lustre-MDT0000: new disk, initializing [10791.385577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10791.408048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10795.776974] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10805.097188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10805.209579] Lustre: 326057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10805.257228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10805.261318] Lustre: Skipped 1 previous similar message [10805.338912] Lustre: lustre-MDT0001: new disk, initializing [10805.432246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10805.477090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10805.501019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10809.047979] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10817.719199] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10822.532262] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10828.045033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10828.326212] Lustre: lustre-OST0000: new disk, initializing [10828.330607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10828.341967] Lustre: 328024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10828.445872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10832.945680] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10835.550693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10835.561113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10835.669201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10842.239618] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10856.421547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10856.424564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.433537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10856.437234] LustreError: Skipped 1 previous similar message [10856.465275] Lustre: Skipped 1 previous similar message [10861.537546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10861.549131] Lustre: Skipped 1 previous similar message [10861.900644] Lustre: server umount lustre-OST0000 complete [10866.660173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.672325] Lustre: Skipped 1 previous similar message [10866.686660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10871.777796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10871.789983] Lustre: Skipped 2 previous similar messages [10872.220268] Lustre: server umount lustre-MDT0000 complete [10875.597117] LustreError: 326050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787042714 with bad export cookie 6715289053032206629 [10875.605526] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10875.612058] LustreError: 326050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10875.973959] Lustre: server umount lustre-MDT0001 complete [10886.027177] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10888.007930] Key type lgssc unregistered [10888.253061] LNet: 329931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10888.262067] LNetError: 329931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10888.276526] LNet: Removed LNI 192.168.201.137@tcp [10888.946900] Key type .llcrypt unregistered [10888.948958] Key type ._llcrypt unregistered [10904.114576] Key type ._llcrypt registered [10904.116658] Key type .llcrypt registered [10904.216726] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10905.334225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10905.370371] alg: No test for adler32 (adler32-zlib) [10906.352611] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [10906.542043] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10908.186079] Key type lgssc registered [10909.193649] Lustre: Echo OBD driver; http://www.lustre.org/ [10915.490988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10915.516853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10916.833396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10920.784169] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10927.081469] LustreError: 331186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.104248] LustreError: 331186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10928.064623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.299270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10932.078764] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10937.895201] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10942.429357] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10948.112515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10948.509699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10953.969379] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10960.268263] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10974.089559] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10975.303584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10979.809249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10979.824418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10979.840167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10982.192578] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [10982.376521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10982.393114] Lustre: Skipped 1 previous similar message [10984.927983] Lustre: server umount lustre-MDT0000 complete [10987.286267] LustreError: 331182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.260368] LustreError: 331166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787042826 with bad export cookie 4236015229324248088 [10988.279864] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10988.286082] LustreError: 331166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10992.396745] LustreError: 331181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10992.419847] LustreError: 331181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10992.424158] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [10992.439845] Lustre: Skipped 3 previous similar messages [10992.612425] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10994.692381] Lustre: server umount lustre-MDT0001 complete [10997.914199] Lustre: server umount lustre-OST0000 complete [11004.751532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.069284] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11010.804288] Lustre: 335364:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.137@tcp [11012.652171] Lustre: server umount MGS complete [11019.551572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.873486] LustreError: 335935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11019.892397] LustreError: 335935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11019.941874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11023.390588] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11025.378928] LustreError: 335936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11030.190217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11030.438312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11034.421841] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11040.507516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11040.699277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11044.376645] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11049.625080] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11050.996983] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11054.975777] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11056.083534] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11060.197939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11060.210947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11060.216111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11060.222102] Lustre: Skipped 3 previous similar messages [11064.632146] Lustre: server umount lustre-OST0000 complete [11070.433459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11070.444024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11070.444812] Lustre: Skipped 2 previous similar messages [11073.405525] Lustre: server umount lustre-MDT0000 complete [11075.554405] LustreError: 335935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.560068] LustreError: 335935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11076.063144] LustreError: 338331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787042914 with bad export cookie 4236015229324248921 [11076.065196] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11076.074907] LustreError: 338331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11076.292772] Lustre: server umount lustre-MDT0001 complete [11084.447315] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11086.342371] Key type lgssc unregistered [11086.563742] LNet: 339369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11086.567936] LNetError: 339369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11087.594291] LNet: Removed LNI 192.168.201.137@tcp [11088.115456] Key type .llcrypt unregistered [11088.117981] Key type ._llcrypt unregistered [11101.302248] Key type ._llcrypt registered [11101.306150] Key type .llcrypt registered [11101.398044] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [11113.863715] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11114.513986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11114.688626] alg: No test for adler32 (adler32-zlib) [11115.633900] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [11115.784892] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [11117.423274] Key type lgssc registered [11118.220447] Lustre: Echo OBD driver; http://www.lustre.org/ [11152.809924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11152.833367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.079529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11154.108817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11154.208458] Lustre: lustre-MDT0000: new disk, initializing [11154.318823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11154.361913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11159.749806] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11177.355853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11177.646842] Lustre: 342523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11177.811302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11177.819339] Lustre: Skipped 1 previous similar message [11177.976805] Lustre: lustre-MDT0001: new disk, initializing [11178.117865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11178.210982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11178.240510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11184.153173] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11194.437679] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11199.856529] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11208.267181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11208.496585] Lustre: lustre-OST0000: new disk, initializing [11208.499528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11208.503912] Lustre: 344492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11208.644245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11216.845410] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11218.005601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11218.014133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11218.178624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11229.179965] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11236.723413] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11237.064992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11243.276759] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11243.506965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11249.120166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11249.124751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.133119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11254.246206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11254.249117] Lustre: Skipped 2 previous similar messages [11259.371308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11259.376752] Lustre: Skipped 1 previous similar message [11260.385157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11260.524941] Lustre: server umount lustre-OST0000 complete [11264.487802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11264.501868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.505505] Lustre: Skipped 2 previous similar messages [11264.524231] Lustre: Skipped 1 previous similar message [11269.601803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.609575] Lustre: Skipped 1 previous similar message [11270.389612] Lustre: server umount lustre-MDT0000 complete [11273.742183] LustreError: 342516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787043112 with bad export cookie 1093225657049016967 [11273.755945] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11274.029791] Lustre: server umount lustre-MDT0001 complete [11281.182877] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:51:59 (1787043119) [11295.612387] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [11304.402612] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11339.427499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11339.592741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11339.617830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11339.705708] Lustre: lustre-MDT0000: new disk, initializing [11339.803249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11339.814164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11343.824937] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11352.955136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11353.064313] Lustre: 349255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11353.090776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11353.093579] Lustre: Skipped 1 previous similar message [11353.191488] Lustre: lustre-MDT0001: new disk, initializing [11353.294991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11353.334777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11353.346610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11357.604612] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11366.327276] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11371.867510] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11378.445897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11378.648570] Lustre: lustre-OST0000: new disk, initializing [11378.652742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11378.658598] Lustre: 351223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11378.755628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11384.380913] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11388.476214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11388.482182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11388.602736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11393.407769] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11419.628247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11419.637056] LustreError: Skipped 1 previous similar message [11419.642408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.654766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11419.660364] Lustre: Skipped 1 previous similar message [11425.217259] Lustre: server umount lustre-OST0000 complete [11429.856938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.878037] Lustre: Skipped 2 previous similar messages [11435.309027] Lustre: server umount lustre-MDT0000 complete [11438.762409] LustreError: 349247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787043277 with bad export cookie 1093225657049017849 [11438.773698] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11438.776014] LustreError: 349247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11439.054575] Lustre: server umount lustre-MDT0001 complete [11449.111918] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11451.540838] Key type lgssc unregistered [11451.811779] LNet: 353374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11451.824351] LNetError: 353374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11452.843664] LNet: Removed LNI 192.168.201.137@tcp [11453.654926] Key type .llcrypt unregistered [11453.656902] Key type ._llcrypt unregistered [11546.185537] Key type ._llcrypt registered [11546.188978] Key type .llcrypt registered [11546.311721] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11546.571295] Key type .llcrypt unregistered [11546.578413] Key type ._llcrypt unregistered [11612.579367] Key type ._llcrypt registered [11612.581307] Key type .llcrypt registered [11612.688143] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11612.889426] Key type .llcrypt unregistered [11612.891170] Key type ._llcrypt unregistered [11638.779506] Key type ._llcrypt registered [11638.781632] Key type .llcrypt registered [11638.849295] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11639.086462] Key type .llcrypt unregistered [11639.088150] Key type ._llcrypt unregistered [11663.801628] Key type ._llcrypt registered [11663.803647] Key type .llcrypt registered [11663.941699] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11664.195596] Key type .llcrypt unregistered [11664.198201] Key type ._llcrypt unregistered [11728.923936] Key type ._llcrypt registered [11728.925586] Key type .llcrypt registered [11729.012176] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11729.267514] Key type .llcrypt unregistered [11729.269763] Key type ._llcrypt unregistered [11753.728958] Key type ._llcrypt registered [11753.730660] Key type .llcrypt registered [11753.811097] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11754.054295] Key type .llcrypt unregistered [11754.056721] Key type ._llcrypt unregistered [11777.570145] Key type ._llcrypt registered [11777.572554] Key type .llcrypt registered [11777.680952] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11777.935555] Key type .llcrypt unregistered [11777.937284] Key type ._llcrypt unregistered [11802.915214] Key type ._llcrypt registered [11802.918773] Key type .llcrypt registered [11802.986151] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11803.242403] Key type .llcrypt unregistered [11803.244875] Key type ._llcrypt unregistered [11831.072360] Key type ._llcrypt registered [11831.077590] Key type .llcrypt registered [11831.167408] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11831.373427] Key type .llcrypt unregistered [11831.374798] Key type ._llcrypt unregistered [11856.061431] Key type ._llcrypt registered [11856.065185] Key type .llcrypt registered [11856.180598] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11856.363283] Key type .llcrypt unregistered [11856.364937] Key type ._llcrypt unregistered [11915.289731] Key type ._llcrypt registered [11915.292915] Key type .llcrypt registered [11915.392273] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11915.625288] Key type .llcrypt unregistered [11915.629248] Key type ._llcrypt unregistered [11946.344404] Key type ._llcrypt registered [11946.346564] Key type .llcrypt registered [11946.516577] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11946.911387] Key type .llcrypt unregistered [11946.914125] Key type ._llcrypt unregistered [11974.175362] Key type ._llcrypt registered [11974.177359] Key type .llcrypt registered [11974.340648] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11974.597702] Key type .llcrypt unregistered [11974.599877] Key type ._llcrypt unregistered [12022.020924] Key type ._llcrypt registered [12022.029608] Key type .llcrypt registered [12022.116830] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [12035.671564] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12036.876737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12036.903675] alg: No test for adler32 (adler32-zlib) [12037.917966] Lustre: Lustre: Build Version: 2.17.55_1_g50984fb [12038.106965] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12039.743171] Key type lgssc registered [12040.888841] Lustre: Echo OBD driver; http://www.lustre.org/ [12072.523840] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12072.548207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.755909] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12073.776308] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12073.822339] Lustre: tst250fs-MDT0000: new disk, initializing [12073.879471] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12073.893439] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12077.794764] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12087.636143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12087.745502] Lustre: 373527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12087.797260] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12087.800386] Lustre: Skipped 1 previous similar message [12087.869307] Lustre: tst250fs-MDT0001: new disk, initializing [12087.932506] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12087.968634] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12087.983319] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12091.546783] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12096.270295] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12101.651490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12101.912870] Lustre: tst250fs-OST0000: new disk, initializing [12101.922940] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12101.933226] Lustre: 375157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12102.043135] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12102.266831] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12102.277300] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12102.298230] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12107.636969] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12116.567746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12116.745207] Lustre: tst250fs-OST0001: new disk, initializing [12116.748422] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12116.753204] Lustre: 376030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12116.846407] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12122.782989] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12124.235082] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12124.247528] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12124.308192] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12127.579273] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12130.155441] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12130.163663] Lustre: Skipped 2 previous similar messages [12149.729488] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12149.735110] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.746537] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12150.244496] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.250056] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12150.272222] Lustre: Skipped 1 previous similar message [12150.274452] Lustre: Skipped 1 previous similar message [12152.909840] Lustre: server umount tst250fs-MDT0000 complete [12155.923985] LustreError: 373521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787043994 with bad export cookie 4109063136552004738 [12155.928460] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12155.939800] LustreError: 373521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12156.243900] Lustre: server umount tst250fs-MDT0001 complete [12169.778385] Lustre: server umount tst250fs-OST0000 complete [12184.027309] Lustre: server umount tst250fs-OST0001 complete [12199.834519] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [12206.590259] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12230.334273] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 05:07:48 (1787044068) [12231.530456] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12244.763682] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [12251.647440] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12274.901532] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11931 sec ========= 05:08:32 (1787044112) [12276.041350] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:08:34 (1787044114) === [12278.451633] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:08:36 (1787044116) === [12309.703603] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12311.748982] Key type lgssc unregistered [12311.996533] LNet: 385156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12312.006811] LNetError: 385156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12313.065388] LNet: Removed LNI 192.168.201.137@tcp [12313.800165] Key type .llcrypt unregistered [12313.802309] Key type ._llcrypt unregistered