[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479935697 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003367] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.006010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021705] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025287] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] *** VALIDATE tmpfs *** [ 0.030330] *** VALIDATE proc *** [ 0.031238] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040131] debug: unmapping init [mem 0xffffffff91059000-0xffffffff91060fff] [ 0.042180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043542] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.051291] rcu: Hierarchical SRCU implementation. [ 0.053292] smp: Bringing up secondary CPUs ... [ 0.054588] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.058223] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142492] node 0 deferred pages initialised in 79ms [ 0.145108] devtmpfs: initialized [ 0.146247] x86/mm: Memory block size: 128MB [ 0.148538] gcov: version magic: 0x41383552 [ 0.151370] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156265] pinctrl core: initialized pinctrl subsystem [ 0.158136] [ 0.158579] ************************************************************* [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163011] ** ** [ 0.164007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166012] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.171011] ** IOMMU data structures, which may compromise security on ** [ 0.173009] ** your system. ** [ 0.174025] ** ** [ 0.177015] ** If you see this message and you are not debugging the ** [ 0.179010] ** kernel, report this immediately to your vendor! ** [ 0.181024] ** ** [ 0.184013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186012] ************************************************************* [ 0.188668] NET: Registered protocol family 16 [ 0.190437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199433] cpuidle: using governor menu [ 0.201684] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205524] PCI: Using configuration type 1 for base access [ 0.209131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228054] cryptd: max_cpu_qlen set to 1000 [ 0.231254] ACPI: Added _OSI(Module Device) [ 0.233015] ACPI: Added _OSI(Processor Device) [ 0.234015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241509] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248138] ACPI: Interpreter enabled [ 0.249062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251013] ACPI: Using IOAPIC for interrupt routing [ 0.253100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256427] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266488] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280314] acpiphp: Slot [2] registered [ 0.282118] acpiphp: Slot [5] registered [ 0.284130] acpiphp: Slot [6] registered [ 0.285126] acpiphp: Slot [7] registered [ 0.286103] acpiphp: Slot [8] registered [ 0.287090] acpiphp: Slot [9] registered [ 0.289151] acpiphp: Slot [10] registered [ 0.291078] acpiphp: Slot [3] registered [ 0.292124] acpiphp: Slot [4] registered [ 0.294121] acpiphp: Slot [11] registered [ 0.296099] acpiphp: Slot [12] registered [ 0.297092] acpiphp: Slot [13] registered [ 0.298103] acpiphp: Slot [14] registered [ 0.300098] acpiphp: Slot [15] registered [ 0.301101] acpiphp: Slot [16] registered [ 0.303122] acpiphp: Slot [17] registered [ 0.305077] acpiphp: Slot [18] registered [ 0.306060] acpiphp: Slot [19] registered [ 0.307096] acpiphp: Slot [20] registered [ 0.308096] acpiphp: Slot [21] registered [ 0.310098] acpiphp: Slot [22] registered [ 0.311102] acpiphp: Slot [23] registered [ 0.312079] acpiphp: Slot [24] registered [ 0.314103] acpiphp: Slot [25] registered [ 0.315114] acpiphp: Slot [26] registered [ 0.317096] acpiphp: Slot [27] registered [ 0.318090] acpiphp: Slot [28] registered [ 0.319107] acpiphp: Slot [29] registered [ 0.321095] acpiphp: Slot [30] registered [ 0.322106] acpiphp: Slot [31] registered [ 0.324082] PCI host bridge to bus 0000:00 [ 0.325016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338682] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340865] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362475] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365939] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372864] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.401861] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.428014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438594] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.446015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482937] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.489015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.498015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.518016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.529117] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.539014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.545013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.567014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.577699] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.601012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.615015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.625149] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.631014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.657014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.667226] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.670274] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.673445] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677418] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.680244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.687095] iommu: Default domain type: Passthrough [ 0.690466] SCSI subsystem initialized [ 0.692118] ACPI: bus type USB registered [ 0.694103] usbcore: registered new interface driver usbfs [ 0.696079] usbcore: registered new interface driver hub [ 0.699112] usbcore: registered new device driver usb [ 0.700167] pps_core: LinuxPPS API ver. 1 registered [ 0.702012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.706053] PTP clock support registered [ 0.709052] EDAC MC: Ver: 3.0.0 [ 0.711128] PCI: Using ACPI for IRQ routing [ 0.714882] NetLabel: Initializing [ 0.716011] NetLabel: domain hash size = 128 [ 0.718010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.720100] NetLabel: unlabeled traffic allowed by default [ 0.722133] vgaarb: loaded [ 0.724264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.727014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.735823] clocksource: Switched to clocksource kvm-clock [ 0.840319] VFS: Disk quotas dquot_6.6.0 [ 0.841750] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.843669] *** VALIDATE ramfs *** [ 0.844807] *** VALIDATE hugetlbfs *** [ 0.846704] pnp: PnP ACPI init [ 0.848419] pnp: PnP ACPI: found 6 devices [ 0.864502] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.868231] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.870716] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.872883] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.875664] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.878510] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.882062] NET: Registered protocol family 2 [ 0.884446] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.888374] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.891326] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.897430] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.900506] TCP: Hash tables configured (established 65536 bind 65536) [ 0.902765] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.905466] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907446] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910174] NET: Registered protocol family 1 [ 0.912504] RPC: Registered named UNIX socket transport module. [ 0.914880] RPC: Registered udp transport module. [ 0.916465] RPC: Registered tcp transport module. [ 0.918224] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.920912] NET: Registered protocol family 44 [ 0.922709] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.925145] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.927704] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.930415] PCI: CLS 0 bytes, default 64 [ 0.933106] Unpacking initramfs... [ 2.319241] debug: unmapping init [mem 0xffff904c3cc54000-0xffff904c3ffbffff] [ 2.324722] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.328299] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.333421] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.817305] Initialise system trusted keyrings [ 2.819175] Key type blacklist registered [ 2.821117] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.829858] zbud: loaded [ 2.832593] *** VALIDATE nfs *** [ 2.833746] *** VALIDATE nfs4 *** [ 2.835171] pstore: using deflate compression [ 2.838701] Platform Keyring initialized [ 2.939375] NET: Registered protocol family 38 [ 2.941141] Key type asymmetric registered [ 2.942771] Asymmetric key parser 'x509' registered [ 2.944691] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.947966] io scheduler mq-deadline registered [ 2.949875] io scheduler kyber registered [ 2.951698] io scheduler bfq registered [ 2.953811] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.957376] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.960618] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.963916] ACPI: Power Button [PWRF] [ 2.968954] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.975690] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.987852] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.995125] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.008758] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.036949] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.069136] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.074714] Non-volatile memory driver v1.3 [ 3.076314] Linux agpgart interface v0.103 [ 3.106019] virtio_blk virtio1: [vda] 149800 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.108880] vda: detected capacity change from 0 to 76697600 [ 3.124213] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.127448] vdb: detected capacity change from 0 to 1073741824 [ 3.139650] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.143030] vdc: detected capacity change from 0 to 2621440000 [ 3.157802] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.161291] vdd: detected capacity change from 0 to 2621440000 [ 3.176394] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.179304] vde: detected capacity change from 0 to 4294967296 [ 3.192729] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.195974] vdf: detected capacity change from 0 to 4294967296 [ 3.202456] libphy: Fixed MDIO Bus: probed [ 3.209518] usbcore: registered new interface driver usbserial_generic [ 3.211642] usbserial: USB Serial support registered for generic [ 3.213599] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.217096] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.218559] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.221217] mousedev: PS/2 mouse device common for all mice [ 3.224632] rtc_cmos 00:05: RTC can wake from S4 [ 3.228094] rtc_cmos 00:05: registered as rtc0 [ 3.228477] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.229601] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.235920] intel_pstate: CPU model not supported [ 3.239646] hid: raw HID events driver (C) Jiri Kosina [ 3.241531] usbcore: registered new interface driver usbhid [ 3.243964] usbhid: USB HID core driver [ 3.246244] drop_monitor: Initializing network drop monitor service [ 3.248501] Initializing XFRM netlink socket [ 3.249714] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.250954] NET: Registered protocol family 10 [ 3.256174] Segment Routing with IPv6 [ 3.256202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.257190] NET: Registered protocol family 17 [ 3.261123] mpls_gso: MPLS GSO support [ 3.266459] RAS: Correctable Errors collector initialized. [ 3.268395] AVX version of gcm_enc/dec engaged. [ 3.269671] AES CTR mode by8 optimization enabled [ 3.328752] sched_clock: Marking stable (3328721530, 0)->(4209097333, -880375803) [ 3.331952] registered taskstats version 1 [ 3.333668] Loading compiled-in X.509 certificates [ 3.335525] zswap: loaded using pool lzo/zbud [ 3.359848] Key type big_key registered [ 3.372255] Key type encrypted registered [ 3.374119] ima: No TPM chip found, activating TPM-bypass! [ 3.376633] ima: Allocated hash algorithm: sha1 [ 3.378233] ima: No architecture policies found [ 3.379480] evm: Initialising EVM extended attributes: [ 3.381000] evm: security.selinux [ 3.382142] evm: security.ima [ 3.383300] evm: security.capability [ 3.384870] evm: HMAC attrs: 0x1 [ 3.387475] rtc_cmos 00:05: setting system clock to 2026-09-04 13:49:13 UTC (1788529753) [ 3.393537] debug: unmapping init [mem 0xffffffff92003000-0xffffffff921fffff] [ 3.397396] debug: unmapping init [mem 0xffffffff90d82000-0xffffffff91058fff] [ 3.406136] Write protecting the kernel read-only data: 28672k [ 3.410431] debug: unmapping init [mem 0xffffffff8f403000-0xffffffff8f5fffff] [ 3.413176] debug: unmapping init [mem 0xffffffff8fd14000-0xffffffff8fdfffff] [ 3.444541] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.453840] systemd[1]: Detected virtualization kvm. [ 3.456372] systemd[1]: Detected architecture x86-64. [ 3.458548] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.484469] systemd[1]: No hostname configured. [ 3.486220] systemd[1]: Set hostname to . [ 3.488118] random: systemd: uninitialized urandom read (16 bytes read) [ 3.490514] systemd[1]: Initializing machine ID from random generator. [ 3.521842] random: ln: uninitialized urandom read (6 bytes read) [ 3.602137] random: systemd: uninitialized urandom read (16 bytes read) [ 3.604840] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.610454] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.614695] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.158419] device-mapper: uevent: version 1.0.3 [ 4.161112] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.804464] virtio_net virtio0 ens2: renamed from eth0 [ 4.815702] random: fast init done [ 4.878287] scsi host0: ata_piix [ 4.944670] scsi host1: ata_piix [ 4.945942] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.947770] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.508831] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.895453] random: crng init done [ 9.897284] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.290640] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.537140] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.812182] SELinux: Disabled at runtime. [ 11.875675] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.887684] systemd[1]: Detected virtualization kvm. [ 11.890086] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.436157] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.440124] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.444045] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.448048] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.452239] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.461716] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.466494] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ 12.633511] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ 12.905265] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.264508] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.335794] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.553352] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.571946] EDAC sbridge: Ver: 1.1.2 [ 15.557776] Key type dns_resolver registered [ 15.875436] NFS: Registering the id_resolver key type [ 15.877970] Key type id_resolver registered [ 15.880427] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg615-server login: [ 42.355868] libcfs: loading out-of-tree module taints kernel. [ 42.378717] Key type ._llcrypt registered [ 42.380366] Key type .llcrypt registered [ 42.433064] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 58.357525] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 60.215592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.242808] alg: No test for adler32 (adler32-zlib) [ 61.943334] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 63.087123] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 64.903217] Key type lgssc registered [ 66.876774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.650019] hrtimer: interrupt took 5008902 ns [ 82.983674] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 123.471911] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 136.748389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.795464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.013323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.042698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.181377] Lustre: lustre-MDT0000: new disk, initializing [ 138.290927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.313867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.694647] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.961509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.042437] Lustre: 6509:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.088875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.095068] Lustre: Skipped 1 previous similar message [ 157.193145] Lustre: lustre-MDT0001: new disk, initializing [ 157.271976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.319985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.327758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.909542] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.502951] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.466513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.918993] Lustre: lustre-OST0000: new disk, initializing [ 176.940132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.961349] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.146983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.492409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.506349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.570940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 185.916989] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.653331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.782804] Lustre: lustre-OST0001: new disk, initializing [ 201.787406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.795570] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.884446] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.057653] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.547345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.558701] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.621309] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.551823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.847798] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.021453] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.014583] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing yml_node [ 247.821109] Lustre: DEBUG MARKER: Client: 2.17.58.5 [ 250.120764] Lustre: DEBUG MARKER: MDS: 2.17.58.5 [ 252.452579] Lustre: DEBUG MARKER: OSS: 2.17.58.5 [ 253.997518] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 09:53:23 EDT 2026 [ 269.552675] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 270.881651] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.739502] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.742596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.756794] Lustre: Skipped 3 previous similar messages [ 288.764392] Lustre: Skipped 3 previous similar messages [ 292.412862] Lustre: server umount lustre-MDT0000 complete [ 296.134550] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530046 with bad export cookie 3345628154547662805 [ 296.139380] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.150871] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 298.979473] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.990743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.997795] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.009710] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 299.019238] Lustre: Skipped 1 previous similar message [ 301.984780] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.985597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.002779] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 302.025915] Lustre: Skipped 2 previous similar messages [ 302.536712] Lustre: server umount lustre-MDT0001 complete [ 306.414734] Lustre: server umount lustre-OST0000 complete [ 310.434472] Lustre: server umount lustre-OST0001 complete [ 317.275678] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 327.809909] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 369.816752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.095140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 370.126632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 370.219192] Lustre: lustre-MDT0000: new disk, initializing [ 370.329327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.361296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 374.952713] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 385.640690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.791256] Lustre: 15073:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 385.887369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 385.890504] Lustre: Skipped 1 previous similar message [ 385.987566] Lustre: lustre-MDT0001: new disk, initializing [ 386.099879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.158114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 386.178876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.374958] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 401.083785] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 406.750518] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 413.608253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.200756] Lustre: lustre-OST0000: new disk, initializing [ 414.224840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.236863] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 414.334415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.350052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.374716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.560173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 421.588835] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 431.872978] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.395090] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 437.644243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.292556] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.493809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.436288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.438196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.440140] LustreError: Skipped 1 previous similar message [ 446.442085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.451618] Lustre: Skipped 1 previous similar message [ 451.564429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.590991] Lustre: Skipped 2 previous similar messages [ 460.255365] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 460.559969] Lustre: server umount lustre-OST0000 complete [ 466.914979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.921913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.944766] Lustre: Skipped 1 previous similar message [ 466.948276] Lustre: Skipped 2 previous similar messages [ 472.045094] LustreError: 18382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.063459] LustreError: 18382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 472.131158] Lustre: server umount lustre-MDT0000 complete [ 475.857212] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530225 with bad export cookie 3345628154547664485 [ 475.859547] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.870338] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 476.156828] Lustre: server umount lustre-MDT0001 complete [ 482.807975] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:57:11 (1788530231) [ 502.167414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.762841] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.850068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.462432] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 507.872773] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.635140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.239046] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 526.763661] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 532.277287] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 534.335637] Lustre: Failing over lustre-MDT0000 [ 534.600398] Lustre: server umount lustre-MDT0000 complete [ 535.937657] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530286 with bad export cookie 3345628154547665409 [ 535.957459] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.970193] Lustre: 21556:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 536.547140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.560627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.588079] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.626063] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 537.004264] Lustre: 21557:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 541.091347] Lustre: 21556:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 546.272655] Lustre: 21557:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 551.400434] Lustre: 21556:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 553.771155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.618963] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 559.585982] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 559.594373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 559.622629] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 567.022119] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 574.949362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 574.953239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.962087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 574.977495] Lustre: Skipped 2 previous similar messages [ 574.993313] Lustre: Skipped 2 previous similar messages [ 585.183167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 585.509314] Lustre: server umount lustre-MDT0000 complete [ 586.219437] LustreError: 21401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.234913] LustreError: 21401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 588.884970] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530338 with bad export cookie 3345628154547665983 [ 588.893210] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.901694] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 589.380771] Lustre: server umount lustre-MDT0001 complete [ 597.851853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.415866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.424206] Lustre: Skipped 2 previous similar messages [ 603.207991] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 612.301460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.063855] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 627.208841] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 632.976245] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 635.437764] Lustre: Failing over lustre-MDT0000 [ 635.926867] Lustre: server umount lustre-MDT0000 complete [ 637.197406] LustreError: 23242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530387 with bad export cookie 3345628154547666417 [ 637.221044] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.242901] Lustre: 24889:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 638.316483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.322337] LustreError: 23256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.330435] Lustre: Skipped 1 previous similar message [ 638.366229] LustreError: 23256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 654.835240] Lustre: 24891:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 654.845863] Lustre: 24891:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 4 previous similar messages [ 655.847762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.216152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.227816] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 658.248481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 658.252756] Lustre: Skipped 1 previous similar message [ 658.268840] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 660.776147] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 670.372710] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 676.833569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.839339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 676.854075] Lustre: Skipped 1 previous similar message [ 676.866426] Lustre: Skipped 4 previous similar messages [ 689.122860] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 689.524972] Lustre: server umount lustre-MDT0000 complete [ 693.501842] LustreError: 23997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530443 with bad export cookie 3345628154547666984 [ 693.502211] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.514304] LustreError: 23997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 693.942981] Lustre: server umount lustre-MDT0001 complete [ 703.318325] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:00:52 (1788530452) [ 722.310512] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 734.582930] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 772.826249] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 783.952411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.210594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 784.250846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 784.315352] Lustre: lustre-MDT0000: new disk, initializing [ 784.379512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.386271] Lustre: Skipped 2 previous similar messages [ 784.397102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 788.582259] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 799.319218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.419818] Lustre: 29653:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 799.741243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 804.456084] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 814.589150] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 820.190802] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 827.121387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 827.414079] Lustre: 31620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 828.563185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 828.604511] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 828.726586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 828.747897] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 833.522755] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 842.844435] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 848.349823] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 869.864410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.865700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 869.890099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 869.907298] Lustre: Skipped 6 previous similar messages [ 873.874217] Lustre: server umount lustre-OST0000 complete [ 884.101480] Lustre: server umount lustre-MDT0000 complete [ 885.223114] LustreError: 29664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.249192] LustreError: 29664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 888.597024] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530638 with bad export cookie 3345628154547667418 [ 888.603401] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 888.615697] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 889.046496] Lustre: server umount lustre-MDT0001 complete [ 900.272086] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 902.899744] Key type lgssc unregistered [ 903.152687] LNet: 33601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.156708] LNetError: 33601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.171829] LNet: Removed LNI 192.168.206.115@tcp [ 903.926229] Key type .llcrypt unregistered [ 903.931771] Key type ._llcrypt unregistered [ 926.651751] Key type ._llcrypt registered [ 926.653901] Key type .llcrypt registered [ 926.782418] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 927.880217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 927.924409] alg: No test for adler32 (adler32-zlib) [ 929.030480] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 929.268502] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 930.967967] Key type lgssc registered [ 932.153692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.724810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 939.758902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.343612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.565127] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 951.781707] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.801130] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 954.296825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.711253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.144841] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 966.677646] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 972.171048] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 979.239150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.430568] LustreError: 36831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.643403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.087131] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 985.522797] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 986.131885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 993.580529] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1007.882969] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:05:57 (1788530757) [ 1009.806861] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1011.559742] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1018.889841] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:06:07 (1788530767) [ 1027.572871] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:06:16 (1788530776) [ 1028.587302] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1037.006485] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1037.011602] Lustre: Skipped 2 previous similar messages [ 1038.717314] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:06:27 (1788530787) [ 1042.162852] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1042.169154] Lustre: Skipped 3 previous similar messages [ 1059.680090] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1059.684124] Lustre: Skipped 6 previous similar messages [ 1063.123142] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:06:51 (1788530811) [ 1065.970199] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1069.525407] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1077.934434] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:07:07 (1788530827) [ 1079.585790] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1079.591028] Lustre: Skipped 3 previous similar messages [ 1092.232868] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:07:21 (1788530841) [ 1107.936621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1107.952401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.990274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.962056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.962210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.968225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.983022] Lustre: Skipped 1 previous similar message [ 1114.084915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.090608] Lustre: Skipped 1 previous similar message [ 1119.207342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.247171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1121.672186] Lustre: server umount lustre-OST0000 complete [ 1128.929389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1128.945240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.975049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.985286] Lustre: Skipped 1 previous similar message [ 1139.685804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.689503] Lustre: Skipped 4 previous similar messages [ 1141.216208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1141.432827] Lustre: server umount lustre-MDT0000 complete [ 1144.801462] LustreError: 36031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.817901] LustreError: 36031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1144.844701] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530894 with bad export cookie 12617939111034503745 [ 1144.849768] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.852090] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1145.075839] Lustre: server umount lustre-MDT0001 complete [ 1155.005517] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1157.664453] Key type lgssc unregistered [ 1157.920905] LNet: 42869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.931627] LNetError: 42869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.956862] LNet: Removed LNI 192.168.206.115@tcp [ 1158.694197] Key type .llcrypt unregistered [ 1158.696343] Key type ._llcrypt unregistered [ 1177.672434] Key type ._llcrypt registered [ 1177.675289] Key type .llcrypt registered [ 1177.818427] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 1192.688621] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1193.689605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.714926] alg: No test for adler32 (adler32-zlib) [ 1194.754930] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1195.041474] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1196.807260] Key type lgssc registered [ 1197.839556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.711446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.739548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.953609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1232.977630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1233.048469] Lustre: lustre-MDT0000: new disk, initializing [ 1233.156351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.174820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1236.587110] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1246.515719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.607741] Lustre: 46018:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1246.706256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1246.710158] Lustre: Skipped 1 previous similar message [ 1246.772666] Lustre: lustre-MDT0001: new disk, initializing [ 1246.861912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.893235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1246.908310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1250.622369] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1259.511217] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1266.066358] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1272.413413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1272.704658] Lustre: lustre-OST0000: new disk, initializing [ 1272.709337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1272.716673] Lustre: 47987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1272.840976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.438759] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1279.577810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1279.589092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1279.714061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1287.937613] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.207911] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1292.367960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1296.673816] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1296.895861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1300.460094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.464108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.470838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.480983] LustreError: Skipped 1 previous similar message [ 1300.496896] Lustre: Skipped 1 previous similar message [ 1305.582740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.592730] Lustre: Skipped 2 previous similar messages [ 1310.690840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.712904] Lustre: Skipped 1 previous similar message [ 1313.247164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1313.479622] Lustre: server umount lustre-OST0000 complete [ 1318.883309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.890834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.911662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1320.930987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.888354] Lustre: server umount lustre-MDT0000 complete [ 1327.451431] LustreError: 46010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531077 with bad export cookie 17418416283567901730 [ 1327.459478] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.467090] LustreError: 46010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1327.898395] Lustre: server umount lustre-MDT0001 complete [ 1337.909835] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:11:26 (1788531086) [ 1344.458718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.872169] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.901539] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1344.935446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.909797] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1350.112382] LustreError: 50452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.214831] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.424324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.823659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1362.098415] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1368.721967] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1373.789803] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1380.166329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1380.546595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.713205] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1394.418311] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1398.635910] LustreError: 52979:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1399.516139] Lustre: Modifying parameter general.timeout=1 in log params [ 1401.092295] Lustre: Modifying parameter general.timeout=3 in log params [ 1401.095623] Lustre: Skipped 1 previous similar message [ 1403.471700] Lustre: Modifying parameter general.timeout=6 in log params [ 1403.475378] Lustre: Skipped 2 previous similar messages [ 1408.084640] Lustre: Modifying parameter general.timeout=12 in log params [ 1408.089294] Lustre: Skipped 5 previous similar messages [ 1416.630223] Lustre: Modifying parameter general.timeout=21 in log params [ 1416.637387] Lustre: Skipped 8 previous similar messages [ 1433.334814] Lustre: Modifying parameter general.timeout=38 in log params [ 1433.341684] Lustre: Skipped 16 previous similar messages [ 1466.114281] Lustre: Modifying parameter general.timeout=73 in log params [ 1466.118331] Lustre: Skipped 34 previous similar messages [ 1524.932986] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:14:33 (1788531273) [ 1528.063816] Lustre: 59927:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1530.234757] Lustre: 60028:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1531.291892] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1531.298294] Lustre: Skipped 60 previous similar messages [ 1532.509521] Lustre: 60127:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1536.857543] Lustre: 60340:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1536.876884] Lustre: 60340:0:(mgs_llog.c:1456:mgs_modify_param()) Skipped 1 previous similar message [ 1565.796884] Lustre: 61812:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1565.806180] Lustre: 61812:0:(mgs_llog.c:1456:mgs_modify_param()) Skipped 1 previous similar message [ 1565.832206] LustreError: 61812:0:(mgs_handler.c:1215:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1569.908487] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:15:19 (1788531319) [ 1580.514973] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.523665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.531657] Lustre: Skipped 1 previous similar message [ 1580.545675] Lustre: Skipped 4 previous similar messages [ 1586.035930] Lustre: server umount lustre-MDT0000 complete [ 1589.919114] LustreError: 50431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531340 with bad export cookie 17418416283567902647 [ 1589.920664] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.927703] LustreError: 50431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1590.295936] Lustre: server umount lustre-MDT0001 complete [ 1604.706791] Lustre: server umount lustre-OST0000 complete [ 1628.977784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.246173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1629.490758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.475146] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1641.817957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.896556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1641.914762] Lustre: 64339:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1646.004585] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1652.101692] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1658.154516] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1664.999799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1665.227439] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1665.736275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.755207] Lustre: Skipped 1 previous similar message [ 1672.757528] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1680.021312] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1681.481725] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1681.490901] Lustre: Skipped 19 previous similar messages [ 1681.534195] Lustre: 66907:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1691.620426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1691.620773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.624894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.624901] Lustre: Skipped 3 previous similar messages [ 1691.626924] LustreError: Skipped 1 previous similar message [ 1691.649324] Lustre: Skipped 2 previous similar messages [ 1693.918916] Lustre: server umount lustre-OST0000 complete [ 1698.794043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1701.858387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.878479] Lustre: Skipped 1 previous similar message [ 1703.588913] Lustre: server umount lustre-MDT0000 complete [ 1706.976464] LustreError: 64346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.987187] LustreError: 64346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1707.162340] LustreError: 65082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531457 with bad export cookie 17418416283567905713 [ 1707.169663] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1707.173869] LustreError: 65082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1707.426182] Lustre: server umount lustre-MDT0001 complete [ 1718.011165] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1720.471303] Key type lgssc unregistered [ 1720.789185] LNet: 68117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.803884] LNetError: 68117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.834942] LNet: Removed LNI 192.168.206.115@tcp [ 1721.575280] Key type .llcrypt unregistered [ 1721.583569] Key type ._llcrypt unregistered [ 1744.937556] Key type ._llcrypt registered [ 1744.940611] Key type .llcrypt registered [ 1745.090873] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1746.337575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.373651] alg: No test for adler32 (adler32-zlib) [ 1747.484153] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1747.744465] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1749.392515] Key type lgssc registered [ 1750.490382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.617414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.631253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.330666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.950468] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1770.466767] LustreError: 69525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1770.514093] LustreError: 69525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1775.267307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.078905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1780.449820] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1790.984389] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1798.477932] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1805.228934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.731718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.210455] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1818.920459] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1825.441299] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1835.491100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.496322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1835.515018] Lustre: Skipped 2 previous similar messages [ 1835.534740] Lustre: Skipped 2 previous similar messages [ 1839.675456] Lustre: server umount lustre-MDT0000 complete [ 1840.611316] LustreError: 69521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1840.623871] LustreError: 69521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1843.311399] LustreError: 69506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531593 with bad export cookie 1554241612122218614 [ 1843.321858] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.322908] LustreError: 69506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1843.653640] Lustre: server umount lustre-MDT0001 complete [ 1857.918250] Lustre: server umount lustre-OST0000 complete [ 1885.587078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.993229] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1886.044027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1886.431948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.787169] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1901.991410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.124217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1902.200870] Lustre: 74397:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1902.735897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1907.370114] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1916.247776] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1923.277093] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1931.013651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.191454] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1931.586564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.047346] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1945.006395] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1946.737458] Lustre: 76971:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1948.570883] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1956.329281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.332724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1956.348591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.777380] Lustre: server umount lustre-OST0000 complete [ 1964.007776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1964.016609] LustreError: Skipped 1 previous similar message [ 1964.025770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.041533] Lustre: Skipped 1 previous similar message [ 1964.048808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.056953] Lustre: Skipped 1 previous similar message [ 1966.560332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.564648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.628854] Lustre: server umount lustre-MDT0000 complete [ 1971.692025] LustreError: 77144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.715981] LustreError: 77144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1972.714959] LustreError: 74388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531722 with bad export cookie 1554241612122219503 [ 1972.717129] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.722034] LustreError: 74388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1973.030394] Lustre: server umount lustre-MDT0001 complete [ 1982.349721] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1984.999659] Key type lgssc unregistered [ 1985.264621] LNet: 78186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.275444] LNetError: 78186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.292218] LNet: Removed LNI 192.168.206.115@tcp [ 1985.955258] Key type .llcrypt unregistered [ 1985.958366] Key type ._llcrypt unregistered [ 2010.107691] Key type ._llcrypt registered [ 2010.111515] Key type .llcrypt registered [ 2010.236898] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2011.730833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.802337] alg: No test for adler32 (adler32-zlib) [ 2012.847325] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 2013.095416] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2014.783287] Key type lgssc registered [ 2015.833965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2023.452748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2023.470370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.047132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.333415] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2035.175394] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.213383] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2037.612175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.222222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.742792] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2049.619571] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2054.151423] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2059.477691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2059.974461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.758361] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2071.718104] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2077.829290] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2078.606410] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2079.927442] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2079.931542] Lustre: Skipped 1 previous similar message [ 2082.321084] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2082.326139] Lustre: Skipped 2 previous similar messages [ 2087.146062] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2087.153094] Lustre: Skipped 5 previous similar messages [ 2095.832662] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2095.840409] Lustre: Skipped 10 previous similar messages [ 2112.013831] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2112.022579] Lustre: Skipped 20 previous similar messages [ 2144.725909] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2144.732480] Lustre: Skipped 37 previous similar messages [ 2166.660503] Lustre: 87684:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2169.371519] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:25:18 (1788531918) [ 2170.830206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2172.645219] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:25:21 (1788531921) [ 2178.016570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.021210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.047650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.068524] Lustre: Skipped 1 previous similar message [ 2178.079417] Lustre: Skipped 1 previous similar message [ 2182.830823] Lustre: server umount lustre-OST0000 complete [ 2186.722792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2186.729570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.749089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.257552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.257880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.278124] Lustre: Skipped 1 previous similar message [ 2192.919130] Lustre: server umount lustre-MDT0000 complete [ 2193.381488] LustreError: 79588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.406394] LustreError: 79588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2196.601271] LustreError: 79575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531946 with bad export cookie 4876923582882248206 [ 2196.608075] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.610146] LustreError: 79575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2197.017134] Lustre: server umount lustre-MDT0001 complete [ 2207.168051] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2209.653443] Key type lgssc unregistered [ 2209.949510] LNet: 88931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.954237] LNetError: 88931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.975930] LNet: Removed LNI 192.168.206.115@tcp [ 2210.747975] Key type .llcrypt unregistered [ 2210.751145] Key type ._llcrypt unregistered [ 2215.035603] Key type ._llcrypt registered [ 2215.042923] Key type .llcrypt registered [ 2215.135508] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2219.381770] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules [ 2220.305511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2220.345160] alg: No test for adler32 (adler32-zlib) [ 2221.392838] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 2221.654926] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2223.351184] Key type lgssc registered [ 2224.485234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2225.614389] LustreError: 90129:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2241.842483] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2248.096078] LustreError: 90129:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2250.829539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2250.850111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.323108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.258970] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2262.507744] LustreError: 91155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.532655] LustreError: 91155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2266.812444] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:26:55 (1788532015) [ 2267.616819] LustreError: 91154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.326017] LustreError: 91186:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2272.335066] LustreError: 91186:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2272.346693] LustreError: 91186:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2273.027624] Lustre: server umount lustre-MDT0000 complete [ 2286.783823] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2289.781845] Key type lgssc unregistered [ 2290.085150] LNet: 92680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.093159] LNetError: 92680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.111207] LNet: Removed LNI 192.168.206.115@tcp [ 2290.940718] Key type .llcrypt unregistered [ 2290.949618] Key type ._llcrypt unregistered [ 2305.453608] Key type ._llcrypt registered [ 2305.457625] Key type .llcrypt registered [ 2305.542680] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2306.343833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2306.444940] alg: No test for adler32 (adler32-zlib) [ 2307.466753] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 2307.648281] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2309.319153] Key type lgssc registered [ 2310.292386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2317.089586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2317.109197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.687505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.472548] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2328.040993] LustreError: 93888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.058236] LustreError: 93888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2330.078472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.797250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2335.717627] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2344.137379] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2350.381403] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2357.114716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2357.771664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.676470] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2370.784133] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2421.370945] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:29:30 (1788532170) [ 2423.046723] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2424.880513] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:29:33 (1788532173) [ 2431.460038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.469556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.479972] Lustre: Skipped 2 previous similar messages [ 2431.498145] Lustre: Skipped 2 previous similar messages [ 2435.428475] Lustre: server umount lustre-MDT0000 complete [ 2436.581460] LustreError: 93882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.605496] LustreError: 93882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2439.513641] LustreError: 97146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532189 with bad export cookie 3846829959585594792 [ 2439.515926] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2439.525243] LustreError: 97146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2439.867067] Lustre: server umount lustre-MDT0001 complete [ 2455.043514] Lustre: server umount lustre-OST0000 complete [ 2467.197738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2467.724087] LustreError: 98178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.929273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.622667] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2472.943439] LustreError: 98179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.025321] LustreError: 98178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.456815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.039545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2485.974800] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2493.426921] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2499.447657] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2512.245878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.414649] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2512.423427] LustreError: 98168:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2512.451645] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2512.458421] LustreError: 100261:0:(tgt_mount.c:2742:server_fill_super()) Unable to start targets: -98 [ 2512.464847] LustreError: 100261:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 2512.471340] LustreError: 100261:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 2512.539375] Lustre: server umount lustre-OST0000 complete [ 2512.543911] LustreError: 100261:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2520.813724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.969149] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2520.978186] LustreError: 98168:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2521.001073] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2521.006365] LustreError: 100727:0:(tgt_mount.c:2742:server_fill_super()) Unable to start targets: -98 [ 2521.010846] LustreError: 100727:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 2521.015329] LustreError: 100727:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 2521.077572] Lustre: server umount lustre-OST0000 complete [ 2521.081567] LustreError: 100727:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2530.752134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2530.860612] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2530.867253] Lustre: Found index 0 for lustre-OST0000, updating log [ 2530.873357] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2530.986855] Lustre: lustre-OST0000: new disk, initializing [ 2530.990655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2530.996654] Lustre: 101264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2531.220716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.032218] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2540.727711] Lustre: Failing over lustre-OST0000 [ 2540.984749] Lustre: server umount lustre-OST0000 complete [ 2547.690236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.702959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.713245] Lustre: Skipped 1 previous similar message [ 2551.842273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.856788] Lustre: Skipped 1 previous similar message [ 2555.016763] Lustre: server umount lustre-MDT0000 complete [ 2556.896649] LustreError: 98179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.914235] LustreError: 98179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2558.477692] LustreError: 100746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532308 with bad export cookie 3846829959585597592 [ 2558.484596] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.490363] LustreError: 100746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2558.960106] Lustre: server umount lustre-MDT0001 complete [ 2567.237075] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:31:56 (1788532316) [ 2584.709383] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2593.959716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.390677] LustreError: 103611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.499084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.739404] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2605.890230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.039045] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2613.564990] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2619.714230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.511719] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2625.528021] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2625.563780] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2625.565700] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2625.588517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2626.625441] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2636.575209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.649814] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2636.821407] Lustre: lustre-OST0001: new disk, initializing [ 2636.828454] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2636.835458] Lustre: 105930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2637.275871] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2637.288164] Lustre: Skipped 2 previous similar messages [ 2644.378708] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2645.316253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2645.332177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2645.499603] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2652.856515] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2663.430381] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2668.000713] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2668.010637] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.021803] Lustre: Skipped 1 previous similar message [ 2668.027298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2671.075570] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.938580] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 2672.949575] Lustre: Skipped 3 previous similar messages [ 2674.272495] Lustre: server umount lustre-MDT0001 complete [ 2676.197629] LustreError: 103611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.207683] LustreError: 103611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2683.829294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.935698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2683.940704] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2683.986178] Lustre: 103600:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2685.544749] LustreError: 107487:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2685.563590] LustreError: 107487:0:(tgt_mount.c:739:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2685.571020] LustreError: 103603:0:(mgc_request.c:718:do_requeue()) failed processing log: -17 [ 2685.820180] LustreError: 107514:0:(obd_config.c:2100:class_config_llog_handler()) MGC192.168.206.115@tcp: cfg command failed: rc = -17 [ 2685.846181] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2685.846181] [ 2688.743735] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2690.035746] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2698.371692] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:34:07 (1788532447) [ 2716.103957] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2734.469845] Lustre: 109549:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2759.182381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2763.644811] Lustre: 110684:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2848.228047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2848.237319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.247143] Lustre: Skipped 1 previous similar message [ 2848.262745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.994307] Lustre: server umount lustre-MDT0000 complete [ 2853.856261] LustreError: 103607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.884188] LustreError: 103607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2856.591174] LustreError: 103592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532606 with bad export cookie 3846829959585598677 [ 2856.595694] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.605587] LustreError: 103592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2856.904815] Lustre: server umount lustre-MDT0001 complete [ 2870.962767] Lustre: server umount lustre-OST0000 complete [ 2884.773138] Lustre: server umount lustre-OST0001 complete [ 2889.640990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2899.252319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2906.608574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2916.238166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2934.944794] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2943.839844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.868932] Lustre: lustre-MDT0000: reset Object Index mappings [ 2944.313038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.317996] Lustre: Skipped 1 previous similar message [ 2948.300291] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2955.929710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.959912] Lustre: lustre-MDT0001: reset Object Index mappings [ 2960.538769] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2963.345585] Lustre: 115782:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2969.220435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.359945] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2981.304193] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2985.047462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2989.000561] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2989.013225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2989.159627] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2992.555854] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3000.643528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3004.407897] Lustre: 117661:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3051.520267] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:40:00 (1788532800) [ 3079.144183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.153306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.168196] Lustre: Skipped 3 previous similar messages [ 3079.173082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.179783] Lustre: Skipped 3 previous similar messages [ 3083.506454] Lustre: server umount lustre-MDT0000 complete [ 3086.306550] LustreError: 114636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.332743] LustreError: 114636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3087.063879] LustreError: 116967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532837 with bad export cookie 3846829959585645955 [ 3087.069743] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.073536] LustreError: 116967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3093.687954] Lustre: server umount lustre-MDT0001 complete [ 3104.317193] Lustre: server umount lustre-OST0000 complete [ 3108.405538] LustreError: 118625:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x37:0x0].0x0 (ffff904cbc5b7600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3114.724282] Lustre: server umount lustre-OST0001 complete [ 3122.456947] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3132.872464] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3169.692849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.973280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3169.996953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3170.089363] Lustre: lustre-MDT0000: new disk, initializing [ 3170.156571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.179940] Lustre: Skipped 3 previous similar messages [ 3170.225865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3174.532795] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3181.609967] Lustre: server umount lustre-MDT0000 complete [ 3192.257462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.624142] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3192.644198] Lustre: 121932:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3192.682452] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3197.798466] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3202.889619] Lustre: server umount lustre-MDT0000 complete [ 3211.370403] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:42:40 (1788532960) [ 3213.180301] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3215.278947] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:42:44 (1788532964) [ 3234.311550] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3244.311037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.195877] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3258.946656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.040763] Lustre: 123983:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3259.123537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3259.126334] Lustre: Skipped 1 previous similar message [ 3259.221040] Lustre: lustre-MDT0001: new disk, initializing [ 3259.356125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3259.374937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3264.840600] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3270.977590] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3279.746082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3280.182348] Lustre: lustre-OST0000: new disk, initializing [ 3280.186769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3280.198061] Lustre: 125562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3282.030402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3282.039552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3282.150449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3286.943216] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3297.131787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.268927] Lustre: lustre-OST0001: new disk, initializing [ 3297.272233] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3297.282865] Lustre: 126432:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3298.730366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3298.737623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3298.910850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3304.576505] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3314.153955] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3317.651602] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3711.199764] Lustre: 132849:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3711.216170] Lustre: 132849:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3711.236698] Lustre: 132849:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3711.255394] Lustre: 132849:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3711.274491] Lustre: 132849:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3711.289569] Lustre: 132849:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4026.782711] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:56:14 (1788533774) [ 4043.232700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4043.247326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4043.266050] Lustre: Skipped 5 previous similar messages [ 4043.282381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.285196] Lustre: Skipped 8 previous similar messages [ 4046.313159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.866263] Lustre: server umount lustre-MDT0000 complete [ 4051.425672] LustreError: 127443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.445733] LustreError: 127443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4053.973040] LustreError: 123975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533804 with bad export cookie 3846829959585713792 [ 4053.985263] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4053.993967] LustreError: 123975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4056.547391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.563712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4056.565126] Lustre: Skipped 3 previous similar messages [ 4056.582064] Lustre: Skipped 5 previous similar messages [ 4060.733277] Lustre: server umount lustre-MDT0001 complete [ 4072.517223] Lustre: server umount lustre-OST0000 complete [ 4091.871183] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4092.060595] Lustre: server umount lustre-OST0001 complete [ 4100.113058] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4111.508212] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4149.125661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.356575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4149.411612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4149.520938] Lustre: lustre-MDT0000: new disk, initializing [ 4149.594147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.598282] Lustre: Skipped 5 previous similar messages [ 4149.613271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4155.308162] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4167.723108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.882305] Lustre: 135914:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4168.055547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4168.059183] Lustre: Skipped 1 previous similar message [ 4168.121579] Lustre: lustre-MDT0001: new disk, initializing [ 4168.262928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4168.279085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4173.315906] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4183.955494] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4191.010512] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4198.893497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4199.437229] Lustre: lustre-OST0000: new disk, initializing [ 4199.446977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4199.456245] Lustre: 137885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4199.621735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4199.625031] Lustre: Skipped 1 previous similar message [ 4200.940386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4200.951788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4201.041787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4206.662982] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4217.765469] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4228.824256] Lustre: lustre-MDD0000: changelog on [ 4231.668786] Lustre: lustre-MDD0001: changelog on [ 6207.700137] Lustre: lustre-MDD0001: changelog off [ 6209.982123] Lustre: 138183:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6211.150563] Lustre: lustre-MDD0000: changelog off [ 6218.210519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.214687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.233709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.233719] Lustre: Skipped 2 previous similar messages [ 6218.262351] Lustre: Skipped 1 previous similar message [ 6223.329730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6223.343171] Lustre: Skipped 2 previous similar messages [ 6228.452496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6228.461921] Lustre: Skipped 1 previous similar message [ 6232.031206] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6232.257607] Lustre: server umount lustre-OST0000 complete [ 6236.641563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6236.651945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.672210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6238.696431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.313711] Lustre: server umount lustre-MDT0000 complete [ 6243.810830] LustreError: 135927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.844577] LustreError: 135927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6247.903931] LustreError: 136720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535998 with bad export cookie 3846829959586534647 [ 6247.912123] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6247.917690] LustreError: 136720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6248.573987] Lustre: server umount lustre-MDT0001 complete [ 6259.506669] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6261.877919] Key type lgssc unregistered [ 6262.146809] LNet: 147042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6262.155285] LNetError: 147042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6262.168534] LNet: Removed LNI 192.168.206.115@tcp [ 6263.003270] Key type .llcrypt unregistered [ 6263.006203] Key type ._llcrypt unregistered [ 6282.650793] Key type ._llcrypt registered [ 6282.657547] Key type .llcrypt registered [ 6282.758628] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6298.047920] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6298.969499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6299.011829] alg: No test for adler32 (adler32-zlib) [ 6300.210721] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 6300.425083] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6302.135119] Key type lgssc registered [ 6303.424178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6340.454584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6340.501728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.777918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6341.813635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6341.907558] Lustre: lustre-MDT0000: new disk, initializing [ 6341.970040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6341.984334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6346.504255] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6356.588606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.662821] Lustre: 150241:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.761640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6356.766688] Lustre: Skipped 1 previous similar message [ 6356.870906] Lustre: lustre-MDT0001: new disk, initializing [ 6356.985135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6357.067961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6357.086937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6361.519346] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6371.073920] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6377.409357] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6385.398491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.760498] Lustre: lustre-OST0000: new disk, initializing [ 6385.767405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6385.785943] Lustre: 152210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6385.898515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.438032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6391.453141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6391.649949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6393.325328] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6403.419921] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6409.566471] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_module obdecho/obdecho [ 6411.567885] LustreError: 153214:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6411.576212] LustreError: 153214:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6413.435336] LustreError: 153311:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6427.635128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.641586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6427.658376] Lustre: Skipped 2 previous similar messages [ 6427.683985] Lustre: Skipped 2 previous similar messages [ 6431.546939] Lustre: server umount lustre-MDT0000 complete [ 6435.620353] LustreError: 151038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536185 with bad export cookie 5326618116858530803 [ 6435.621584] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6435.633183] LustreError: 151038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6436.143688] Lustre: server umount lustre-MDT0001 complete [ 6450.970548] Lustre: server umount lustre-OST0000 complete [ 6474.343198] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6486.327363] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6523.272111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.523837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6523.543633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6523.621270] Lustre: lustre-MDT0000: new disk, initializing [ 6523.684871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.703259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6528.246290] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6540.887358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.021463] Lustre: 157027:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6541.166908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6541.173102] Lustre: Skipped 1 previous similar message [ 6541.284303] Lustre: lustre-MDT0001: new disk, initializing [ 6541.518092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6541.579050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6541.600277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6546.211722] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6555.771660] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6561.610888] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6569.083768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6569.360196] Lustre: lustre-OST0000: new disk, initializing [ 6569.363852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6569.369934] Lustre: 158994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6569.466216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6575.596918] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6579.869453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6579.884512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6580.007344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6585.039374] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6589.807892] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6590.041121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6594.643339] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6594.840860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6600.672023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6600.672303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.688156] LustreError: Skipped 1 previous similar message [ 6600.697384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6603.238616] Lustre: server umount lustre-OST0000 complete [ 6607.841295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6607.845713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.853904] Lustre: Skipped 1 previous similar message [ 6607.860196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6607.864107] Lustre: Skipped 1 previous similar message [ 6610.919108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.920707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.032359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.040211] Lustre: Skipped 2 previous similar messages [ 6622.175953] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6622.626183] Lustre: server umount lustre-MDT0000 complete [ 6625.251721] LustreError: 157033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.288587] LustreError: 157033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6627.282526] LustreError: 157020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536377 with bad export cookie 5326618116858531804 [ 6627.292112] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.298949] LustreError: 157020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6627.845454] Lustre: server umount lustre-MDT0001 complete [ 6632.283893] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:39:41 (1788536381) [ 6635.411560] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6638.347648] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:39:46 (1788536386) [ 6647.253678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.682385] LustreError: 161412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.757531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6652.904065] LustreError: 161413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.214393] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6657.064706] LustreError: 161412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6662.113707] LustreError: 161413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6665.788911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.410918] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6682.709384] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6689.106170] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6696.184548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.502383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.515713] Lustre: Skipped 1 previous similar message [ 6703.375354] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6711.973084] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6722.535262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6722.540818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.551584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6722.555527] Lustre: Skipped 2 previous similar messages [ 6724.799455] Lustre: server umount lustre-MDT0000 complete [ 6727.152274] LustreError: 161408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.177710] LustreError: 161408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6728.302339] LustreError: 161393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536478 with bad export cookie 5326618116858532728 [ 6728.302339] LustreError: 161394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536478 with bad export cookie 5326618116858532728 [ 6728.303484] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6728.313143] LustreError: 161394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6732.257107] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.269620] Lustre: Skipped 2 previous similar messages [ 6734.775267] Lustre: server umount lustre-MDT0001 complete [ 6740.735399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6754.130345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.882113] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6758.901423] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6773.219967] Lustre: 165261:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6774.559554] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6774.571399] Lustre: 165262:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6779.875564] Lustre: 165262:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6784.479330] Lustre: 161404:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.206.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 6784.992580] Lustre: 165262:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6790.117359] Lustre: 165262:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6791.137055] LustreError: 165267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.150428] LustreError: 165267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6799.840254] Lustre: 165261:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6799.858400] Lustre: 165261:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 2 previous similar messages [ 6811.167670] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6820.325693] Lustre: 165262:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6820.347137] Lustre: 165262:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 5 previous similar messages [ 6831.199150] Lustre: 161404:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.206.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 6835.169065] Lustre: 165256:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.206.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 6853.088293] Lustre: 165263:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6853.101861] Lustre: 165263:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 6 previous similar messages [ 6855.238277] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6855.260350] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 47 previous similar messages [ 6857.823195] Lustre: 161404:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.206.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 6875.357596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6877.862329] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6879.950466] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6889.766205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.631469] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6896.109263] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6904.917482] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6907.431647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6914.026837] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6915.329877] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6941.668093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.673072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6941.688273] Lustre: Skipped 2 previous similar messages [ 6941.695477] Lustre: Skipped 6 previous similar messages [ 6945.375224] Lustre: server umount lustre-MDT0000 complete [ 6948.955803] LustreError: 161394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536699 with bad export cookie 5326618116858539371 [ 6948.958437] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6948.969403] LustreError: 161394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6949.436920] Lustre: server umount lustre-MDT0001 complete [ 6963.291848] Lustre: server umount lustre-OST0000 complete [ 6985.714977] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7001.690511] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7045.789015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.000284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7046.034124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7046.161321] Lustre: lustre-MDT0000: new disk, initializing [ 7046.291547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7046.295822] Lustre: Skipped 1 previous similar message [ 7046.319864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7051.965972] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7064.450792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.525613] Lustre: 170398:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7064.774061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7069.290309] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7080.043580] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7086.641261] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7094.155041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7094.444190] Lustre: lustre-OST0000: new disk, initializing [ 7094.448753] Lustre: Skipped 1 previous similar message [ 7094.453872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7094.464409] Lustre: Skipped 2 previous similar messages [ 7094.474623] Lustre: 172367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7101.031173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7101.041876] Lustre: Skipped 1 previous similar message [ 7101.055151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7101.169627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7101.430810] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7112.253233] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7117.191387] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7117.403275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7122.723521] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7122.930260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7127.010708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.011611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7127.035190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7127.044141] Lustre: Skipped 1 previous similar message [ 7139.295167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7139.432111] Lustre: server umount lustre-OST0000 complete [ 7146.993798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7149.597746] Lustre: server umount lustre-MDT0000 complete [ 7152.613709] LustreError: 170409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.633142] LustreError: 170409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 30 previous similar messages [ 7153.636125] LustreError: 170391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536903 with bad export cookie 5326618116858556262 [ 7153.637985] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7153.649552] LustreError: 170391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7154.128518] Lustre: server umount lustre-MDT0001 complete [ 7164.890240] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:48:33 (1788536913) [ 7171.990861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.465532] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7186.235539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.134456] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7199.921632] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7206.140863] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7214.274431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.830840] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7229.831531] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7243.448834] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:49:52 (1788536992) [ 7251.460492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7251.468771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.468781] Lustre: Skipped 3 previous similar messages [ 7251.473352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7251.506310] Lustre: Skipped 8 previous similar messages [ 7254.726547] Lustre: server umount lustre-OST0000 complete [ 7265.391912] Lustre: server umount lustre-MDT0000 complete [ 7269.471055] LustreError: 174760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537019 with bad export cookie 5326618116858557186 [ 7269.485321] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7269.493651] LustreError: 174760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7269.880643] Lustre: server umount lustre-MDT0001 complete [ 7284.118497] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7286.899450] Key type lgssc unregistered [ 7287.155819] LNet: 178511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7287.173924] LNetError: 178511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7287.189769] LNet: Removed LNI 192.168.206.115@tcp [ 7288.269687] Key type .llcrypt unregistered [ 7288.277471] Key type ._llcrypt unregistered [ 7310.114691] Key type ._llcrypt registered [ 7310.119349] Key type .llcrypt registered [ 7310.254198] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7311.553495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7311.594573] alg: No test for adler32 (adler32-zlib) [ 7312.674166] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 7312.863556] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7314.559254] Key type lgssc registered [ 7315.740630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7324.459124] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7324.491458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7325.683303] Lustre: 179766:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7341.215454] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7346.335411] LustreError: 179752:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.206.115@tcp: failed processing log, type 4: rc = -110 [ 7350.752673] Lustre: 179767:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7351.458169] LustreError: 179752:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7351.493867] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7351.502789] LustreError: MGC192.168.206.115@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7351.532581] LustreError: 179752:0:(tgt_mount.c:1948:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7351.549463] LustreError: 179752:0:(tgt_mount.c:2742:server_fill_super()) Unable to start targets: -22 [ 7351.565335] LustreError: 179752:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 7351.709063] Lustre: server umount lustre-OST0000 complete [ 7351.712420] LustreError: 179752:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7363.805723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.228074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.109688] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7374.817693] LustreError: 180256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.841181] LustreError: 180256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7378.258543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.774459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.568694] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7392.016565] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7399.174689] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7407.796302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.920426] LustreError: 182071:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7407.926722] Lustre: 182071:0:(mgc_request_server.c:733:mgc_llog_local_copy()) MGC192.168.206.115@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7408.044159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.143556] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7424.234273] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7428.581938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.591925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7428.599406] Lustre: Skipped 1 previous similar message [ 7428.616417] Lustre: Skipped 2 previous similar messages [ 7433.699421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.719798] Lustre: Skipped 2 previous similar messages [ 7438.816595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.824604] Lustre: Skipped 2 previous similar messages [ 7441.376037] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7441.629985] Lustre: server umount lustre-MDT0000 complete [ 7443.939069] LustreError: 180256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.961975] LustreError: 180256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7446.003600] LustreError: 180237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537196 with bad export cookie 1768698298213415552 [ 7446.016102] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7446.018471] LustreError: 180237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7446.572963] Lustre: server umount lustre-MDT0001 complete [ 7461.659879] Lustre: server umount lustre-OST0000 complete [ 7472.451502] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7485.544061] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7529.468450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.654136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7529.702186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7529.813516] Lustre: lustre-MDT0000: new disk, initializing [ 7529.956080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.979615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7535.032225] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7545.695343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.829721] Lustre: 185603:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7545.965205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7545.972958] Lustre: Skipped 1 previous similar message [ 7546.075911] Lustre: lustre-MDT0001: new disk, initializing [ 7546.196278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7546.241546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7546.258810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7550.902909] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7559.615466] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7565.827426] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7574.296872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7574.587161] Lustre: lustre-OST0000: new disk, initializing [ 7574.599613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7574.607795] Lustre: 187572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7574.714836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.474201] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7583.909083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7583.920027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7584.089893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7592.091775] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7597.327223] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7597.572643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7602.856859] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7603.146310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7609.824857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7609.831909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.841414] LustreError: Skipped 1 previous similar message [ 7609.849472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.870194] Lustre: Skipped 2 previous similar messages [ 7609.908168] Lustre: Skipped 1 previous similar message [ 7611.717943] Lustre: server umount lustre-OST0000 complete [ 7618.016685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7618.024937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.050877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7620.066844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.304112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7630.311303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7630.335955] Lustre: Skipped 5 previous similar messages [ 7630.616403] Lustre: server umount lustre-MDT0000 complete [ 7634.622145] LustreError: 185596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537384 with bad export cookie 1768698298213416329 [ 7634.635125] LustreError: 185596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7634.637187] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.938777] Lustre: server umount lustre-MDT0001 complete [ 7645.360696] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:56:34 (1788537394) [ 7664.610190] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7667.066838] Key type lgssc unregistered [ 7667.441645] LNet: 190522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7667.451479] LNetError: 190522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7667.469917] LNet: Removed LNI 192.168.206.115@tcp [ 7668.326973] Key type .llcrypt unregistered [ 7668.332305] Key type ._llcrypt unregistered [ 7687.121500] Key type ._llcrypt registered [ 7687.128568] Key type .llcrypt registered [ 7687.217786] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7688.707761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7688.758566] alg: No test for adler32 (adler32-zlib) [ 7689.878437] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 7690.199387] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7691.943611] Key type lgssc registered [ 7693.406690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7701.797355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7701.838719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.373617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7708.442291] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7713.764958] LustreError: 191737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.785377] LustreError: 191737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7718.008730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.487219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7723.452705] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7731.691655] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7738.468928] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7746.865918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7747.149241] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7747.156047] LustreError: 193602:0:(llog.c:1696:llog_backup()) MGC192.168.206.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7747.164787] Lustre: 193602:0:(mgc_request_server.c:730:mgc_llog_local_copy()) MGC192.168.206.115@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7747.295233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.468080] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7762.946630] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7769.058919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.064747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7769.088453] Lustre: Skipped 1 previous similar message [ 7769.113262] Lustre: Skipped 2 previous similar messages [ 7771.496914] Lustre: server umount lustre-MDT0000 complete [ 7774.185767] LustreError: 192466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.207643] LustreError: 192466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7775.027500] LustreError: 191716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537525 with bad export cookie 13126309901912735699 [ 7775.031531] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7775.039591] LustreError: 191716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7775.372842] Lustre: server umount lustre-MDT0001 complete [ 7789.233674] Lustre: server umount lustre-OST0000 complete [ 7798.423524] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7812.397608] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7848.335325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.545323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7848.567386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7848.620871] Lustre: lustre-MDT0000: new disk, initializing [ 7848.669835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7848.684226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7852.262889] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7862.350764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7862.459765] Lustre: 197128:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7862.546103] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7862.548667] Lustre: Skipped 1 previous similar message [ 7862.554261] LustreError: 197910:0:(llog.c:1696:llog_backup()) MGC192.168.206.115@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7862.568954] LustreError: 197910:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7862.578803] Lustre: 197910:0:(mgc_request_server.c:753:mgc_llog_local_copy()) MGC192.168.206.115@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7862.613719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7862.622113] Lustre: Skipped 1 previous similar message [ 7862.699973] Lustre: lustre-MDT0001: new disk, initializing [ 7862.786138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7862.822225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7862.834154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7866.733733] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7876.021522] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7881.498350] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7889.063274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7889.397688] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7889.401730] Lustre: Skipped 1 previous similar message [ 7889.407263] LustreError: 199073:0:(llog.c:1696:llog_backup()) MGC192.168.206.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7889.417558] LustreError: 199073:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7889.426455] Lustre: 199073:0:(mgc_request_server.c:753:mgc_llog_local_copy()) MGC192.168.206.115@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7889.445430] Lustre: 199073:0:(mgc_request_server.c:753:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7889.479543] Lustre: lustre-OST0000: new disk, initializing [ 7889.494887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7889.504074] Lustre: 199097:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7889.645913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7896.948238] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7899.131198] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7899.135623] Lustre: Skipped 1 previous similar message [ 7899.147361] LustreError: 197131:0:(llog.c:1696:llog_backup()) MGC192.168.206.115@tcp: failed to backup log lustre-client: rc = -28 [ 7899.174334] LustreError: 197131:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7899.205170] Lustre: 197131:0:(mgc_request_server.c:753:mgc_llog_local_copy()) MGC192.168.206.115@tcp: failed to copy new config lustre-client: rc = -28 [ 7899.223962] Lustre: 197131:0:(mgc_request_server.c:753:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7899.454597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7899.492514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7899.633582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7908.350510] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7913.635841] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7913.861133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7919.539630] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7919.838725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7925.219663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7925.224967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7925.232610] LustreError: Skipped 1 previous similar message [ 7925.236958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7925.244681] Lustre: Skipped 2 previous similar messages [ 7925.262321] Lustre: Skipped 1 previous similar message [ 7928.363562] Lustre: server umount lustre-OST0000 complete [ 7934.436663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7934.463815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.501447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.563226] Lustre: server umount lustre-MDT0000 complete [ 7940.577351] LustreError: 197139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.616080] LustreError: 197139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7944.307866] LustreError: 197120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537694 with bad export cookie 13126309901912736476 [ 7944.326913] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.810293] Lustre: server umount lustre-MDT0001 complete [ 7956.382795] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:01:45 (1788537705) [ 7972.840696] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7983.348287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7983.716368] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.790511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.975918] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7989.218550] LustreError: 202239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.437762] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.457926] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7998.907710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.119978] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8007.008261] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8014.728982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8022.155243] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8032.618901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8032.846638] Lustre: lustre-OST0001: new disk, initializing [ 8032.850991] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8032.861193] Lustre: 204547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8032.969113] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8032.976202] Lustre: Skipped 2 previous similar messages [ 8038.660051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8038.671739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8038.785818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8039.701938] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8050.642954] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8059.550131] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8062.016610] Lustre: DEBUG MARKER: ADD OST3 [ 8068.714958] loop: module loaded [ 8068.826061] loop0: detected capacity change from 0 to 204800000 [ 8069.321760] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8069.341453] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8069.365634] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8077.263794] Lustre: DEBUG MARKER: STOP OST3 [ 8077.449164] loop0: detected capacity change from 0 to 204800000 [ 8084.725636] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8085.015754] Lustre: lustre-OST0003: new disk, initializing [ 8085.020389] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8085.028378] Lustre: 206600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8086.526735] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8086.778316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8086.808446] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8086.872934] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8088.195557] Lustre: Failing over lustre-OST0003 [ 8088.520606] Lustre: server umount lustre-OST0003 complete [ 8088.552416] LustreError: 206707:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8092.147318] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8092.152760] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8092.166580] LustreError: 203726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8092.176361] Lustre: Skipped 2 previous similar messages [ 8093.177712] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8101.125213] Lustre: DEBUG MARKER: START OST3 again [ 8107.987565] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8108.332767] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8108.343875] Lustre: Skipped 1 previous similar message [ 8108.363358] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8110.023811] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8110.174507] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8110.175616] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8115.143345] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8136.742611] Lustre: Failing over lustre-OST0003 [ 8136.934627] Lustre: server umount lustre-OST0003 complete [ 8138.240274] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.252573] LustreError: 208668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.252853] Lustre: Skipped 1 previous similar message [ 8138.291381] LustreError: 208668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8158.176596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8158.185741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.198860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.204824] Lustre: Skipped 2 previous similar messages [ 8162.227189] Lustre: server umount lustre-MDT0000 complete [ 8166.552062] LustreError: 202218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537916 with bad export cookie 13126309901912737400 [ 8166.576711] LustreError: 202218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8166.582245] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8168.933077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8168.947522] Lustre: Skipped 4 previous similar messages [ 8173.208301] Lustre: server umount lustre-MDT0001 complete [ 8177.398393] Lustre: server umount lustre-OST0000 complete [ 8181.762155] Lustre: server umount lustre-OST0001 complete [ 8190.343330] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8202.368592] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8243.097848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.368481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8243.419741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8243.544312] Lustre: lustre-MDT0000: new disk, initializing [ 8243.636656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.658688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8248.527470] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8259.391882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.486822] Lustre: 212010:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8259.584188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8259.587896] Lustre: Skipped 1 previous similar message [ 8259.643500] Lustre: lustre-MDT0001: new disk, initializing [ 8259.806467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8259.816442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8264.964845] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8274.553504] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8281.294785] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.241307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.738384] Lustre: lustre-OST0000: new disk, initializing [ 8291.746788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8291.752754] Lustre: 213978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8292.997642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8293.018875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8293.258314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8302.659406] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8313.366485] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8318.368097] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8318.539853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8322.702775] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8322.950640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8329.196197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8329.198187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.213473] LustreError: Skipped 1 previous similar message [ 8329.215170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8329.230534] Lustre: Skipped 6 previous similar messages [ 8329.234942] Lustre: Skipped 2 previous similar messages [ 8331.150586] Lustre: server umount lustre-OST0000 complete [ 8349.152639] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8349.381082] Lustre: server umount lustre-MDT0000 complete [ 8349.665974] LustreError: 212020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8349.677115] LustreError: 212020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8353.635336] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538103 with bad export cookie 13126309901912740032 [ 8353.650936] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8353.652137] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8354.055509] Lustre: server umount lustre-MDT0001 complete [ 8358.474970] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:08:27 (1788538107) [ 8380.667026] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8391.812841] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8429.695658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.904481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8429.924558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8429.977478] Lustre: lustre-MDT0000: new disk, initializing [ 8430.060621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8434.576666] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8444.911190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.013399] Lustre: 218593:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8445.337894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8450.490250] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8461.265819] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8468.472734] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8476.751569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.127993] Lustre: 220563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8478.690327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8478.882177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8486.478753] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8497.746435] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8503.147349] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8503.377382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8508.502196] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8508.798486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8514.530626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8514.530963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.534777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8514.534784] Lustre: Skipped 6 previous similar messages [ 8514.540164] LustreError: Skipped 1 previous similar message [ 8514.548141] Lustre: Skipped 3 previous similar messages [ 8525.280268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8525.405918] Lustre: server umount lustre-OST0000 complete [ 8535.009267] Lustre: server umount lustre-MDT0000 complete [ 8535.010249] LustreError: 218600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.034330] LustreError: 218600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8538.342529] LustreError: 220562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538288 with bad export cookie 13126309901912740956 [ 8538.343516] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8538.350526] LustreError: 220562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8538.677906] Lustre: server umount lustre-MDT0001 complete [ 8546.462817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8546.967236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.976817] Lustre: Skipped 5 previous similar messages [ 8551.488371] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8560.503065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.288964] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8573.305668] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8578.985603] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8586.099945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8591.811842] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8599.476460] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8610.283658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8610.290375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8610.292986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8610.292994] LustreError: Skipped 1 previous similar message [ 8610.317275] Lustre: Skipped 3 previous similar messages [ 8610.333130] Lustre: Skipped 8 previous similar messages [ 8611.418276] Lustre: server umount lustre-OST0000 complete [ 8621.007741] Lustre: server umount lustre-MDT0000 complete [ 8625.707884] LustreError: 223519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538375 with bad export cookie 13126309901912741880 [ 8625.715174] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8625.733487] LustreError: 223519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8626.277121] Lustre: server umount lustre-MDT0001 complete [ 8741.761649] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8746.169452] Key type lgssc unregistered [ 8746.658785] LNet: 226446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8746.664476] LNetError: 226446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8746.694621] LNet: Removed LNI 192.168.206.115@tcp [ 8748.220178] Key type .llcrypt unregistered [ 8748.228318] Key type ._llcrypt unregistered [ 8774.983763] Key type ._llcrypt registered [ 8774.985713] Key type .llcrypt registered [ 8775.062339] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8788.816932] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8789.843932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8789.900263] alg: No test for adler32 (adler32-zlib) [ 8791.027784] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 8791.285468] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8792.983610] Key type lgssc registered [ 8794.220935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8830.293920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8830.343333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.714622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8831.769075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8831.913803] Lustre: lustre-MDT0000: new disk, initializing [ 8832.022674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8832.036985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8837.425767] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8849.211686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8849.378389] Lustre: 229795:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8849.536198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8849.547311] Lustre: Skipped 1 previous similar message [ 8849.715996] Lustre: lustre-MDT0001: new disk, initializing [ 8849.840380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8849.885154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8849.897917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8856.286860] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8866.425442] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8872.837691] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8879.091571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8879.314150] Lustre: lustre-OST0000: new disk, initializing [ 8879.316927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8879.332532] Lustre: 231764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8879.428654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8885.565837] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8887.426117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8887.438394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8887.525066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8895.194506] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8900.438623] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8900.704450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8905.835861] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8906.057834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8908.264040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8908.276266] Lustre: Skipped 1 previous similar message [ 8908.276373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8908.276738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8908.298380] Lustre: Skipped 1 previous similar message [ 8913.379028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8913.389299] Lustre: Skipped 1 previous similar message [ 8918.507586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8918.522450] Lustre: Skipped 1 previous similar message [ 8922.079175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8922.257017] Lustre: server umount lustre-OST0000 complete [ 8926.689536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8926.700125] LustreError: Skipped 1 previous similar message [ 8926.709776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8926.724957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.738870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.541309] Lustre: server umount lustre-MDT0000 complete [ 8936.555453] LustreError: 229787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538686 with bad export cookie 2193792793887482161 [ 8936.560848] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8936.562499] LustreError: 229787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8937.091495] Lustre: server umount lustre-MDT0001 complete [ 8955.257827] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8967.070348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.671546] LustreError: 234700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8967.709444] LustreError: 234700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8967.765431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8972.770640] LustreError: 234701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.007646] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8977.905076] LustreError: 234700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8983.011985] LustreError: 234701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8983.961961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8984.313418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.123225] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8993.510798] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9000.292652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.699752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9006.806069] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9015.647554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9015.853660] Lustre: lustre-OST0001: new disk, initializing [ 9015.858558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9015.865517] Lustre: 237009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9021.929700] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9023.656726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9023.674849] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9023.788087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9034.035083] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9040.732175] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9053.478958] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:20:02 (1788538802) [ 9056.228435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9056.234915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.252282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9056.262726] Lustre: Skipped 2 previous similar messages [ 9062.136651] Lustre: server umount lustre-MDT0000 complete [ 9064.935864] LustreError: 234695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.966476] LustreError: 234695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9065.932270] LustreError: 234680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538816 with bad export cookie 2193792793887483085 [ 9065.946742] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9065.951697] LustreError: 234680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9066.390748] Lustre: server umount lustre-MDT0001 complete [ 9080.913885] Lustre: server umount lustre-OST0000 complete [ 9095.501728] Lustre: server umount lustre-OST0001 complete [ 9104.179599] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9116.312137] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9155.215235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9155.614093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9155.653650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9155.725290] Lustre: lustre-MDT0000: new disk, initializing [ 9155.786827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9155.793479] Lustre: Skipped 1 previous similar message [ 9155.806385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9161.722908] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9174.622788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9174.750692] Lustre: 240952:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9174.865595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9174.874568] Lustre: Skipped 1 previous similar message [ 9174.978258] Lustre: lustre-MDT0001: new disk, initializing [ 9175.176518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9175.209184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9180.118915] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9191.260388] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9197.575834] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9204.689888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9204.988605] Lustre: lustre-OST0000: new disk, initializing [ 9204.993504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9204.998714] Lustre: 242924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9205.105819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9205.110874] Lustre: Skipped 1 previous similar message [ 9206.931477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9206.937754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9207.082258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9211.357396] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9221.395244] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9226.335617] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9226.552643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9231.897538] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9232.244048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9237.987766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.989436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9238.001710] Lustre: Skipped 3 previous similar messages [ 9238.004930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9238.008829] LustreError: Skipped 1 previous similar message [ 9238.023289] Lustre: Skipped 5 previous similar messages [ 9240.577785] Lustre: server umount lustre-OST0000 complete [ 9246.691740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9246.707775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9246.727874] Lustre: Skipped 1 previous similar message [ 9250.435970] Lustre: server umount lustre-MDT0000 complete [ 9253.345559] LustreError: 240965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9253.368744] LustreError: 240965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9253.966264] LustreError: 240944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539004 with bad export cookie 2193792793887485388 [ 9253.971189] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.978207] LustreError: 240944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9254.314690] Lustre: server umount lustre-MDT0001 complete [ 9261.833941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.274906] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9272.805419] LustreError: 245141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9272.826081] LustreError: 245141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9276.325425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.603507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9276.609262] Lustre: Skipped 1 previous similar message [ 9281.336614] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9288.712429] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9294.026708] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9299.941712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9305.643202] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9312.126497] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9460.193055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9460.202668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9460.215649] Lustre: Skipped 1 previous similar message [ 9460.230332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9460.239869] Lustre: Skipped 3 previous similar messages [ 9462.932926] Lustre: server umount lustre-OST0000 complete [ 9473.047364] Lustre: server umount lustre-MDT0000 complete [ 9476.577352] LustreError: 247437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.592288] LustreError: 247437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9476.795947] LustreError: 245123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539226 with bad export cookie 2193792793887486312 [ 9476.809619] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9477.117235] Lustre: server umount lustre-MDT0001 complete [ 9579.954699] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9582.357597] Key type lgssc unregistered [ 9582.597886] LNet: 248746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9582.605131] LNetError: 248746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9582.627206] LNet: Removed LNI 192.168.206.115@tcp [ 9583.571377] Key type .llcrypt unregistered [ 9583.573408] Key type ._llcrypt unregistered [ 9607.442173] Key type ._llcrypt registered [ 9607.444066] Key type .llcrypt registered [ 9607.527799] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9624.416424] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9625.393877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9625.413579] alg: No test for adler32 (adler32-zlib) [ 9626.414154] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 9626.648738] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9628.343294] Key type lgssc registered [ 9629.434079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9672.447787] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9681.827720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9681.852977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.063773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9683.118563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9683.244261] Lustre: lustre-MDT0000: new disk, initializing [ 9683.324205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9683.342813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9687.664491] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9698.314595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.458559] Lustre: 252815:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9698.566182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9698.571613] Lustre: Skipped 1 previous similar message [ 9698.677871] Lustre: lustre-MDT0001: new disk, initializing [ 9698.809846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.850605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9698.866915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9703.800516] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9708.808928] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9715.947771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9716.285420] Lustre: lustre-OST0000: new disk, initializing [ 9716.291408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9716.301257] Lustre: 254450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9716.418840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9723.053423] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9725.520633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9725.539030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9725.619329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9733.932735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9734.082513] Lustre: lustre-OST0001: new disk, initializing [ 9734.085705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9734.093685] Lustre: 255321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9734.180971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9739.845851] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9742.564358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9742.576442] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9742.695459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9749.495213] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9759.398326] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9770.469146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9770.480662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9770.501934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9771.957203] Lustre: server umount lustre-MDT0000 complete [ 9773.541924] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9775.776521] LustreError: 253611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539525 with bad export cookie 7916653197997346045 [ 9775.794348] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9776.111419] Lustre: server umount lustre-MDT0001 complete [ 9790.522917] Lustre: server umount lustre-OST0000 complete [ 9794.016598] Lustre: 250324:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788539528/real 1788539528] req@ffff904cbd579c00 x1875419464810240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788539544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9794.057925] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9794.080793] Lustre: Skipped 2 previous similar messages [ 9794.563994] Lustre: server umount lustre-OST0001 complete [ 9809.062439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9820.242819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9828.987665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9839.683228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9860.396984] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9870.290551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.304167] Lustre: lustre-MDT0000: reset Object Index mappings [ 9870.560879] LustreError: 260329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9870.575990] LustreError: 260329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9870.691078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.817756] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9875.937427] LustreError: 260330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9880.032723] LustreError: 260329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9883.513146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.546873] Lustre: lustre-MDT0001: reset Object Index mappings [ 9884.095657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9888.511568] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9891.335851] Lustre: 261470:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9897.802515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9904.340626] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9909.479895] LustreError: 261825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.985038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9912.225415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.235039] Lustre: Skipped 1 previous similar message [ 9915.322908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9915.354090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9918.047721] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9925.358716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9929.019620] Lustre: 263344:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9935.328287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9935.343786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9935.371408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.357653] Lustre: server umount lustre-MDT0000 complete [ 9940.734831] LustreError: 260312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539690 with bad export cookie 7916653197997352989 [ 9940.740087] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9940.755208] LustreError: 260312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9940.962214] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.965308] LustreError: 260326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9940.969730] Lustre: Skipped 3 previous similar messages [ 9940.992141] LustreError: 260326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9941.087748] Lustre: server umount lustre-MDT0001 complete [ 9947.857654] Lustre: server umount lustre-OST0000 complete [ 9951.839795] Lustre: server umount lustre-OST0001 complete [ 9971.440039] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:35:20 (1788539720) [ 9987.220965] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9999.962507] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10037.673753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10038.034399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10038.059447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10038.126627] Lustre: lustre-MDT0000: new disk, initializing [10038.180811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10038.196697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10042.732852] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10053.406653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.527505] Lustre: 267597:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10053.679783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10053.691865] Lustre: Skipped 1 previous similar message [10053.810892] Lustre: lustre-MDT0001: new disk, initializing [10054.026622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10054.040968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10059.458675] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10069.760238] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10076.096304] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10083.505190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.882183] Lustre: lustre-OST0000: new disk, initializing [10083.890736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10083.904756] Lustre: 269567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10085.170054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10085.183093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10085.361117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10090.782352] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10100.917955] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10105.775619] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10105.926759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10110.367657] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10110.596792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10116.070693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10116.075589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10116.086373] Lustre: Skipped 1 previous similar message [10116.096851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10116.096859] Lustre: Skipped 3 previous similar messages [10118.783386] Lustre: server umount lustre-OST0000 complete [10125.793564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10125.802960] LustreError: Skipped 1 previous similar message [10125.816382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10125.822285] Lustre: Skipped 1 previous similar message [10129.569222] Lustre: server umount lustre-MDT0000 complete [10131.425219] LustreError: 269894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.443671] LustreError: 269894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10133.223038] LustreError: 267589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539883 with bad export cookie 7916653197997355194 [10133.228027] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10133.233258] LustreError: 267589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10133.544787] Lustre: server umount lustre-MDT0001 complete [10151.079041] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10160.611537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.950696] LustreError: 272499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.985247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10160.991648] Lustre: Skipped 2 previous similar messages [10165.009184] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10172.741673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.568635] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10180.300791] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10186.441226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10192.860426] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10201.720299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10201.869383] Lustre: lustre-OST0001: new disk, initializing [10201.874553] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10201.879698] Lustre: 274806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10203.948094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10203.965994] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10204.147266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10208.647127] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10218.576120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10222.449760] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10227.493223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10227.532175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10238.131796] Lustre: server umount lustre-MDT0000 complete [10239.975658] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.979292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10239.988309] LustreError: 272496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10239.988323] LustreError: 272496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10239.996157] Lustre: Skipped 4 previous similar messages [10242.212414] LustreError: 272482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539992 with bad export cookie 7916653197997356118 [10242.223788] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10242.696532] Lustre: server umount lustre-MDT0001 complete [10256.886815] Lustre: server umount lustre-OST0000 complete [10271.480454] Lustre: server umount lustre-OST0001 complete [10289.827574] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10301.477909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10301.886937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10301.894051] Lustre: Skipped 3 previous similar messages [10307.041591] LustreError: 278152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.068038] LustreError: 278152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10307.144574] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10317.319046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.772055] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10326.307496] Lustre: 279292:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10332.696312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10339.387407] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10346.363855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10346.865791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10352.147561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10353.141126] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10361.011623] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10365.174670] Lustre: 281172:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10370.626221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10370.632667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10384.049574] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:42:12 (1788540132) [10399.712164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10399.736804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10399.748448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10399.755464] Lustre: Skipped 2 previous similar messages [10404.907656] Lustre: server umount lustre-MDT0000 complete [10407.879709] LustreError: 279294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540157 with bad export cookie 7916653197997362621 [10407.884230] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10407.887116] LustreError: 279294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10408.421181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10408.434516] Lustre: Skipped 5 previous similar messages [10422.239173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10422.428292] Lustre: server umount lustre-MDT0001 complete [10440.159380] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10440.388759] Lustre: server umount lustre-OST0000 complete [10444.916911] Lustre: server umount lustre-OST0001 complete [10451.959440] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [10464.401524] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10512.501899] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10526.516527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10526.870352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10526.911530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10527.026867] Lustre: lustre-MDT0000: new disk, initializing [10527.122822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10532.868530] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10546.524939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10546.682931] Lustre: 285699:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10547.014372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10552.419512] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10565.847605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10566.299387] Lustre: 287335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10566.417084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10566.433082] Lustre: Skipped 5 previous similar messages [10567.749957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10573.450989] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10586.128052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10586.352131] Lustre: 288205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10588.480580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10588.488045] Lustre: Skipped 1 previous similar message [10588.721145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10594.118672] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10604.710673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10608.497913] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10608.504557] Lustre: Skipped 1 previous similar message [10620.906781] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:46:10 (1788540370) [10623.969542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10623.983583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.010119] Lustre: Skipped 5 previous similar messages [10624.035771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10624.048713] Lustre: Skipped 6 previous similar messages [10628.890702] Lustre: server umount lustre-MDT0000 complete [10629.606419] LustreError: 285713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.643207] LustreError: 285713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10632.335724] LustreError: 285693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540382 with bad export cookie 7916653197997370671 [10632.345134] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10632.356137] LustreError: 285693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10632.728252] Lustre: server umount lustre-MDT0001 complete [10646.456139] Lustre: server umount lustre-OST0000 complete [10659.095285] Lustre: server umount lustre-OST0001 complete [10666.961907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.703960] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10680.293827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.007956] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10692.508133] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10698.018510] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10704.731862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10711.159064] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10718.050663] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10733.761283] loop1: detected capacity change from 0 to 204800000 [10734.106995] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10734.114600] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10734.125567] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10739.627929] loop1: detected capacity change from 0 to 204800000 [10747.795471] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.874886] LustreError: 290446:0:(mgs_llog.c:680:mgs_find_or_make_fsdb_nolock()) fs1771: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10747.898247] LustreError: 290446:0:(mgs_llog.c:739:mgs_check_index()) Can't get db for fs1771 [10747.902540] LustreError: 290446:0:(mgs_handler.c:517:mgs_target_reg()) fs1771-MDT0001: Error checking target index: rc = -13 [10747.910426] LustreError: fs1771-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10747.928870] LustreError: 294175:0:(tgt_mount.c:2742:server_fill_super()) Unable to start targets: -13 [10747.939227] LustreError: 294175:0:(tgt_mount.c:2192:server_put_super()) no obd fs1771-MDT0001 [10747.947990] LustreError: 294175:0:(tgt_mount.c:130:server_deregister_mount()) fs1771-MDT0001 not registered [10747.977659] Lustre: server umount fs1771-MDT0001 complete [10747.984789] LustreError: 294175:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10756.834752] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10757.072165] Lustre: Setting parameter fs1771-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1771-MDT0001 [10757.172557] Lustre: srv-fs1771-MDT0001: No data found on store. Initialize space. [10757.182841] Lustre: Skipped 4 previous similar messages [10757.302341] Lustre: fs1771-MDT0001: new disk, initializing [10757.305171] Lustre: Skipped 3 previous similar messages [10757.481244] Lustre: srv-fs1771-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10757.485456] Lustre: 294643:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1771-MDT0001-mdtlov: can't create new llog: rc = -115 [10757.501648] LustreError: 294643:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1771-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10761.986157] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10769.429600] Lustre: server umount fs1771-MDT0001 complete [10782.020709] loop1: detected capacity change from 0 to 204800000 [10787.452484] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10787.811971] Lustre: 296211:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1771-MDT0001-mdtlov: can't create new llog: rc = -115 [10787.823564] LustreError: 296211:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1771-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10789.362830] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10792.138355] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10797.246255] Lustre: server umount fs1771-MDT0001 complete [10808.573781] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [10808.584134] Lustre: Skipped 4 previous similar messages [10808.801739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10808.817687] Lustre: Skipped 3 previous similar messages [10821.087156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10821.319924] Lustre: server umount lustre-MDT0000 complete [10829.209197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.292601] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10834.267880] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10834.941524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10841.188809] loop1: detected capacity change from 0 to 204800000 [10846.540967] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.925614] Lustre: 298656:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1771-MDT0001-mdtlov: can't create new llog: rc = -115 [10846.937623] LustreError: 298656:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1771-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10851.346978] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10856.062529] Lustre: server umount fs1771-MDT0001 complete [10880.480849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10880.495446] LustreError: Skipped 2 previous similar messages [10880.503171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10880.510913] Lustre: Skipped 12 previous similar messages [10884.014445] Lustre: server umount lustre-OST0000 complete [10894.401694] Lustre: server umount lustre-MDT0000 complete [10896.364252] LustreError: 290458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.392950] LustreError: 290458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 107 previous similar messages [10898.272391] LustreError: 290439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540648 with bad export cookie 7916653197997373534 [10898.278185] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.283957] LustreError: 290439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10898.645622] Lustre: server umount lustre-MDT0001 complete [10909.456226] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [10912.211100] Key type lgssc unregistered [10912.481787] LNet: 301000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10912.506145] LNetError: 301000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10913.574537] LNet: Removed LNI 192.168.206.115@tcp [10914.393393] Key type .llcrypt unregistered [10914.397211] Key type ._llcrypt unregistered [10931.505954] Key type ._llcrypt registered [10931.508260] Key type .llcrypt registered [10931.582888] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10932.759593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10932.789205] alg: No test for adler32 (adler32-zlib) [10933.781775] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [10934.000635] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [10935.671694] Key type lgssc registered [10936.705995] Lustre: Echo OBD driver; http://www.lustre.org/ [10943.880791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10943.902384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10945.433693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10949.397975] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10955.745419] LustreError: 302261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.757725] LustreError: 302261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10957.339236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.650789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10962.052083] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10968.956585] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10975.606266] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10985.033887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10985.724941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10990.827270] LustreError: 304090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10992.888787] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10996.198484] LustreError: 304091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.624532] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11001.319348] LustreError: 304092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.687996] LustreError: 304090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.720077] LustreError: 304090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11016.929832] LustreError: 304092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.949677] LustreError: 304092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11021.623426] loop1: detected capacity change from 0 to 204800000 [11021.777353] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11021.787870] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11021.797554] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11027.124996] loop1: detected capacity change from 0 to 204800000 [11033.074419] LustreError: 304090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11033.104469] LustreError: 304090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11033.826566] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11033.987506] Lustre: fs19615-OST0000: new disk, initializing [11033.996578] Lustre: srv-fs19615-OST0000: No data found on store. Initialize space. [11034.012277] Lustre: 305996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11034.140451] Lustre: fs19615-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11039.757779] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11048.738447] Lustre: server umount fs19615-OST0000 complete [11060.516552] loop1: detected capacity change from 0 to 204800000 [11065.508320] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11065.772704] Lustre: fs19615-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11068.138935] LustreError: 304090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.158531] LustreError: 304090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11068.911325] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11071.847391] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11080.928158] loop2: detected capacity change from 0 to 204800000 [11081.176966] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11081.202774] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11081.221781] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11084.263650] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11086.828138] loop2: detected capacity change from 0 to 204800000 [11092.605905] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11092.669154] LustreError: fs19615-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11092.689508] LustreError: fs19615-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11092.703045] LustreError: 309037:0:(tgt_mount.c:2742:server_fill_super()) Unable to start targets: -13 [11092.707634] LustreError: 309037:0:(tgt_mount.c:2192:server_put_super()) no obd fs19615-OST0001 [11092.716140] LustreError: 309037:0:(tgt_mount.c:130:server_deregister_mount()) fs19615-OST0001 not registered [11092.749873] Lustre: server umount fs19615-OST0001 complete [11092.753077] LustreError: 309037:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11103.775779] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11103.986390] Lustre: fs19615-OST0001: new disk, initializing [11103.992952] Lustre: srv-fs19615-OST0001: No data found on store. Initialize space. [11104.001406] Lustre: 309590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11104.117608] Lustre: fs19615-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11111.980681] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11121.201679] Lustre: server umount fs19615-OST0000 complete [11131.187634] Lustre: server umount fs19615-OST0001 complete [11134.710201] LustreError: 304312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11134.743927] LustreError: 304312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11143.936915] loop1: detected capacity change from 0 to 204800000 [11149.687407] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11149.913613] Lustre: fs19615-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11156.245768] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11163.223743] loop2: detected capacity change from 0 to 204800000 [11168.937204] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11176.688917] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11187.250409] Lustre: server umount fs19615-OST0000 complete [11197.445428] Lustre: server umount fs19615-OST0001 complete [11212.260151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11212.270580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11212.280668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11212.286582] Lustre: Skipped 1 previous similar message [11217.393286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11217.396991] Lustre: Skipped 1 previous similar message [11217.704849] Lustre: server umount lustre-OST0000 complete [11222.501038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11222.515653] Lustre: Skipped 1 previous similar message [11222.521179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11222.528713] Lustre: Skipped 1 previous similar message [11227.575055] Lustre: server umount lustre-MDT0000 complete [11231.028205] LustreError: 302243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540981 with bad export cookie 15144658939481245951 [11231.038677] LustreError: 302243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11231.042293] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11231.405826] Lustre: server umount lustre-MDT0001 complete [11241.457877] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11244.643551] Key type lgssc unregistered [11244.986518] LNet: 315512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11244.996502] LNetError: 315512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11245.011171] LNet: Removed LNI 192.168.206.115@tcp [11245.802157] Key type .llcrypt unregistered [11245.805215] Key type ._llcrypt unregistered [11281.241308] Key type ._llcrypt registered [11281.244472] Key type .llcrypt registered [11281.386608] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [11296.651268] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [11297.893332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11297.922843] alg: No test for adler32 (adler32-zlib) [11298.917054] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [11299.189797] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [11300.919165] Key type lgssc registered [11301.921523] Lustre: Echo OBD driver; http://www.lustre.org/ [11341.189326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11341.232299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.709633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11342.794318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11342.952786] Lustre: lustre-MDT0000: new disk, initializing [11343.106850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11343.162967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11348.953777] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11359.970571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.076755] Lustre: 319371:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11360.152794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11360.156258] Lustre: Skipped 1 previous similar message [11360.226195] Lustre: lustre-MDT0001: new disk, initializing [11360.310722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11360.352746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11360.368860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11364.560158] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11372.627909] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11377.834100] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11383.951500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11384.198533] Lustre: lustre-OST0000: new disk, initializing [11384.203414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11384.210765] Lustre: 321340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11384.337765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11390.538092] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11393.633335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11393.653324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11393.778465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11399.462077] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11401.221670] Lustre: Mounted lustre-client - version 2.17.58_5_ge0db421 [11402.042832] Lustre: Unmounted lustre-client [11411.430610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11411.432327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11411.447813] Lustre: Skipped 1 previous similar message [11411.458485] Lustre: Skipped 1 previous similar message [11416.230452] Lustre: server umount lustre-OST0000 complete [11421.664924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11421.672483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.680318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11421.690476] Lustre: Skipped 1 previous similar message [11421.703815] Lustre: Skipped 1 previous similar message [11425.946761] Lustre: server umount lustre-MDT0000 complete [11429.673303] LustreError: 319363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541179 with bad export cookie 5991272877449742167 [11429.677607] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11429.691385] LustreError: 319363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11430.099053] Lustre: server umount lustre-MDT0001 complete [11442.477158] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11445.900714] Key type lgssc unregistered [11455.903489] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [11456.608429] LNet: 323501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11456.634593] LNetError: 323501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11531.429702] LNet: Removed LNI 192.168.206.115@tcp [11532.298678] Key type .llcrypt unregistered [11532.301243] Key type ._llcrypt unregistered [11551.410974] Key type ._llcrypt registered [11551.413370] Key type .llcrypt registered [11551.546919] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [11569.832629] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [11570.902433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11570.990668] alg: No test for adler32 (adler32-zlib) [11572.218877] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [11572.490081] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [11574.303591] Key type lgssc registered [11575.701326] Lustre: Echo OBD driver; http://www.lustre.org/ [11613.476246] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:02:42 (1788541362) [11621.007720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11621.037508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.446405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11622.507238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11622.669783] Lustre: lustre-MDT0000: new disk, initializing [11622.773974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11622.793382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11628.320994] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11639.770358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.890926] Lustre: 326733:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11639.943480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11639.947224] Lustre: Skipped 1 previous similar message [11640.008776] Lustre: lustre-MDT0001: new disk, initializing [11640.074933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11640.096490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11640.102945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11646.452330] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11659.003175] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11666.136210] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11674.643576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11674.958750] Lustre: lustre-OST0000: new disk, initializing [11674.963053] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11674.973991] Lustre: 328704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11675.094324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11681.420320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11681.432234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11681.594948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11682.215250] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11693.221611] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11710.439382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.456501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11710.469157] Lustre: Skipped 1 previous similar message [11715.049606] Lustre: server umount lustre-OST0000 complete [11720.672987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11720.685896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11720.687218] Lustre: Skipped 2 previous similar messages [11720.697202] Lustre: Skipped 1 previous similar message [11725.064160] Lustre: server umount lustre-MDT0000 complete [11728.944801] LustreError: 326724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541479 with bad export cookie 16035167453268321912 [11728.954905] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11728.960186] LustreError: 326724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11729.263893] Lustre: server umount lustre-MDT0001 complete [11740.857792] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11745.077723] Key type lgssc unregistered [11745.448619] LNet: 330611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11745.454708] LNetError: 330611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11745.482249] LNet: Removed LNI 192.168.206.115@tcp [11746.320780] Key type .llcrypt unregistered [11746.323953] Key type ._llcrypt unregistered [11773.924334] Key type ._llcrypt registered [11773.926158] Key type .llcrypt registered [11774.100527] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [11775.016215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11775.232574] alg: No test for adler32 (adler32-zlib) [11776.282964] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [11776.478739] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [11778.119235] Key type lgssc registered [11779.338421] Lustre: Echo OBD driver; http://www.lustre.org/ [11788.852923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11788.878891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11790.443242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11795.223106] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11800.040495] LustreError: 331882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.050630] LustreError: 331882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11804.163595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11804.587465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11809.259237] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11817.536441] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11823.844626] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11830.526435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.957843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11837.072545] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11845.538043] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11859.681463] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11861.401120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11866.089264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11866.098122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11866.123986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11868.432255] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [11870.179384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11870.179829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11870.211243] Lustre: Skipped 1 previous similar message [11872.075583] Lustre: server umount lustre-MDT0000 complete [11873.507347] LustreError: 333721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.305037] LustreError: 331882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.322358] LustreError: 331882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11876.134799] LustreError: 331862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541626 with bad export cookie 15796435425099108 [11876.153500] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11876.154632] LustreError: 331862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11878.628681] LustreError: 331877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11878.646924] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [11878.650403] Lustre: Skipped 2 previous similar messages [11880.428494] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11882.655144] Lustre: server umount lustre-MDT0001 complete [11886.797244] Lustre: server umount lustre-OST0000 complete [11895.911503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.872802] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11904.141919] Lustre: 336064:0:(mgs_llog.c:1814:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.115@tcp [11906.690692] Lustre: server umount MGS complete [11916.759409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.273931] LustreError: 336636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.298392] LustreError: 336636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11917.374356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11922.419450] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11927.528978] LustreError: 336637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.559802] LustreError: 336637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11932.790067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11933.175100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11937.787914] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11946.869819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11947.260592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11954.188721] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11963.530585] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11965.457148] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11972.731527] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11974.385450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11979.232033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11979.239643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.265756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11979.272593] Lustre: Skipped 2 previous similar messages [11981.413468] Lustre: server umount lustre-OST0000 complete [11988.460470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11988.462048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11988.470684] Lustre: Skipped 2 previous similar messages [11988.484841] Lustre: Skipped 2 previous similar messages [11991.870533] Lustre: server umount lustre-MDT0000 complete [11993.568279] LustreError: 336637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.584226] LustreError: 336637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11995.579218] LustreError: 336617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541745 with bad export cookie 15796435425100004 [11995.588709] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11995.603667] LustreError: 336617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11995.928516] Lustre: server umount lustre-MDT0001 complete [12007.276866] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12010.038388] Key type lgssc unregistered [12010.352293] LNet: 340071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12010.360451] LNetError: 340071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12010.395329] LNet: Removed LNI 192.168.206.115@tcp [12011.236286] Key type .llcrypt unregistered [12011.238636] Key type ._llcrypt unregistered [12031.731476] Key type ._llcrypt registered [12031.734325] Key type .llcrypt registered [12031.847352] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [12050.052980] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [12051.424146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12051.539102] alg: No test for adler32 (adler32-zlib) [12052.848986] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [12053.092147] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [12054.727174] Key type lgssc registered [12055.904465] Lustre: Echo OBD driver; http://www.lustre.org/ [12091.759805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12091.782294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12093.142503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12093.198347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12093.354139] Lustre: lustre-MDT0000: new disk, initializing [12093.542728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12093.573863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12097.784084] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12108.557658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12108.639300] Lustre: 343230:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12108.708970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12108.713183] Lustre: Skipped 1 previous similar message [12108.776672] Lustre: lustre-MDT0001: new disk, initializing [12108.856429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12108.890966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12108.905253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12113.158804] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12122.039286] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12128.534339] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12136.292161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12136.667437] Lustre: lustre-OST0000: new disk, initializing [12136.674119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12136.685578] Lustre: 345196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12136.814838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12142.173874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12142.192631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12142.364743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12144.851516] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12155.178370] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12161.057146] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12161.364222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12167.459352] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12167.630720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12173.280312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12173.282607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12173.301598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12173.348590] Lustre: Skipped 1 previous similar message [12176.121908] Lustre: server umount lustre-OST0000 complete [12180.449289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12180.465491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12180.489456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12180.498712] Lustre: Skipped 1 previous similar message [12183.523957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12183.525618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12183.561584] Lustre: Skipped 1 previous similar message [12186.703922] Lustre: server umount lustre-MDT0000 complete [12191.025327] LustreError: 343221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541941 with bad export cookie 16411807279621200551 [12191.030718] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12191.049687] LustreError: 343221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12191.530305] Lustre: server umount lustre-MDT0001 complete [12200.456897] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:12:29 (1788541949) [12218.323492] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [12227.947589] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [12264.359926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12264.663882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12264.748469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12264.887475] Lustre: lustre-MDT0000: new disk, initializing [12265.045524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12265.080782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12269.692679] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12280.209817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12280.315764] Lustre: 349958:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12280.374897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12280.378538] Lustre: Skipped 1 previous similar message [12280.452565] Lustre: lustre-MDT0001: new disk, initializing [12280.566413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12280.615757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12280.626269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12285.283658] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12294.299154] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12299.644887] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12306.748366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12307.101590] Lustre: lustre-OST0000: new disk, initializing [12307.106312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12307.112277] Lustre: 351927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12307.202230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12313.736285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12313.753398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12313.862227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12314.270448] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12324.488507] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12365.287092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12365.289877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12365.305053] LustreError: Skipped 1 previous similar message [12365.305355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12365.366928] Lustre: Skipped 1 previous similar message [12366.929934] Lustre: server umount lustre-OST0000 complete [12372.964483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12372.969744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12372.987415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12372.996135] Lustre: Skipped 1 previous similar message [12377.346967] Lustre: server umount lustre-MDT0000 complete [12380.662171] LustreError: 349970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12380.680152] LustreError: 349970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12381.402051] LustreError: 349951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542131 with bad export cookie 16411807279621201475 [12381.405066] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12381.417782] LustreError: 349951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12381.856357] Lustre: server umount lustre-MDT0001 complete [12391.132810] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12393.511977] Key type lgssc unregistered [12393.855318] LNet: 354078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12393.866572] LNetError: 354078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12393.893820] LNet: Removed LNI 192.168.206.115@tcp [12394.745524] Key type .llcrypt unregistered [12394.751959] Key type ._llcrypt unregistered [12491.273588] Key type ._llcrypt registered [12491.278428] Key type .llcrypt registered [12491.396553] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12491.753990] Key type .llcrypt unregistered [12491.756402] Key type ._llcrypt unregistered [12572.374619] Key type ._llcrypt registered [12572.376819] Key type .llcrypt registered [12572.520726] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12572.860384] Key type .llcrypt unregistered [12572.862895] Key type ._llcrypt unregistered [12602.947605] Key type ._llcrypt registered [12602.950152] Key type .llcrypt registered [12603.058673] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12603.332212] Key type .llcrypt unregistered [12603.334800] Key type ._llcrypt unregistered [12636.213952] Key type ._llcrypt registered [12636.218505] Key type .llcrypt registered [12636.358710] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12636.739284] Key type .llcrypt unregistered [12636.743259] Key type ._llcrypt unregistered [12719.273666] Key type ._llcrypt registered [12719.275080] Key type .llcrypt registered [12719.413980] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12719.764702] Key type .llcrypt unregistered [12719.769440] Key type ._llcrypt unregistered [12750.932239] Key type ._llcrypt registered [12750.935486] Key type .llcrypt registered [12751.089096] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12751.368359] Key type .llcrypt unregistered [12751.370411] Key type ._llcrypt unregistered [12784.119564] Key type ._llcrypt registered [12784.125448] Key type .llcrypt registered [12784.233911] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12784.488413] Key type .llcrypt unregistered [12784.491224] Key type ._llcrypt unregistered [12813.126370] Key type ._llcrypt registered [12813.127703] Key type .llcrypt registered [12813.229527] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12813.584378] Key type .llcrypt unregistered [12813.587229] Key type ._llcrypt unregistered [12845.966420] Key type ._llcrypt registered [12845.967798] Key type .llcrypt registered [12846.118989] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12846.413726] Key type .llcrypt unregistered [12846.422977] Key type ._llcrypt unregistered [12877.978568] Key type ._llcrypt registered [12877.983389] Key type .llcrypt registered [12878.082744] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12878.344624] Key type .llcrypt unregistered [12878.347939] Key type ._llcrypt unregistered [12933.792659] Key type ._llcrypt registered [12933.799652] Key type .llcrypt registered [12933.916710] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12934.227425] Key type .llcrypt unregistered [12934.230264] Key type ._llcrypt unregistered [12966.335376] Key type ._llcrypt registered [12966.337384] Key type .llcrypt registered [12966.477782] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12966.750544] Key type .llcrypt unregistered [12966.754288] Key type ._llcrypt unregistered [12997.139091] Key type ._llcrypt registered [12997.142528] Key type .llcrypt registered [12997.236031] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12997.469604] Key type .llcrypt unregistered [12997.472167] Key type ._llcrypt unregistered [13049.337336] Key type ._llcrypt registered [13049.342762] Key type .llcrypt registered [13049.490653] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [13064.501924] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [13065.415016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13065.549103] alg: No test for adler32 (adler32-zlib) [13066.690353] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [13066.891843] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [13068.639170] Key type lgssc registered [13069.827700] Lustre: Echo OBD driver; http://www.lustre.org/ [13109.726039] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13109.770724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13111.177971] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13111.223616] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13111.318809] Lustre: tst250fs-MDT0000: new disk, initializing [13111.385236] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13111.405934] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13117.467680] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13128.902468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13129.007865] Lustre: 374336:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13129.086358] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13129.091600] Lustre: Skipped 1 previous similar message [13129.171918] Lustre: tst250fs-MDT0001: new disk, initializing [13129.300959] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13129.335840] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13129.354124] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13133.356400] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13138.331330] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13145.804468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13146.042303] Lustre: tst250fs-OST0000: new disk, initializing [13146.045397] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13146.049389] Lustre: 375967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13146.151020] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13152.855298] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13154.485739] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13154.510404] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13154.645607] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13163.721568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13163.945076] Lustre: tst250fs-OST0001: new disk, initializing [13163.951363] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13163.958774] Lustre: 376840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13164.119467] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13170.880412] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13170.890212] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13171.019881] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13171.209919] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13176.814778] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13179.413450] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13179.420119] Lustre: Skipped 2 previous similar messages [13200.882690] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13200.893282] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13200.926729] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13201.895729] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13201.900535] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13201.912530] Lustre: Skipped 2 previous similar messages [13201.939256] Lustre: Skipped 3 previous similar messages [13208.430392] Lustre: server umount tst250fs-MDT0000 complete [13212.129864] LustreError: 374347:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13212.156474] LustreError: 374347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [13212.921500] LustreError: 374328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542963 with bad export cookie 12558433847068669175 [13212.928398] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13212.940051] LustreError: 374328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13213.279714] Lustre: server umount tst250fs-MDT0001 complete [13227.264493] Lustre: server umount tst250fs-OST0000 complete [13240.436357] Lustre: server umount tst250fs-OST0001 complete [13259.995492] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [13268.960184] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [13300.924746] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:30:50 (1788543050) [13302.472733] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13320.539935] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [13332.903379] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [13366.433441] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13111 sec ========= 13:31:55 (1788543115) [13368.217501] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:31:57 (1788543117) === [13371.682447] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:32:00 (1788543120) === [13410.376940] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [13412.807291] Key type lgssc unregistered [13413.022826] LNet: 385999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13413.031506] LNetError: 385999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13413.056175] LNet: Removed LNI 192.168.206.115@tcp [13413.900346] Key type .llcrypt unregistered [13413.902582] Key type ._llcrypt unregistered