[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 930984242 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003379] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.010466] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.011000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012013] pid_max: default: 32768 minimum: 301 [ 0.013133] LSM: Security Framework initializing [ 0.014065] Yama: becoming mindful. [ 0.015050] SELinux: Initializing. [ 0.017166] *** VALIDATE selinux *** [ 0.032444] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.043610] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.044267] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.046166] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.047114] *** VALIDATE tmpfs *** [ 0.051259] *** VALIDATE proc *** [ 0.052458] *** VALIDATE cgroup *** [ 0.053015] *** VALIDATE cgroup2 *** [ 0.054326] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.055276] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.056013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.057029] Spectre V2 : User space: Vulnerable [ 0.058012] Speculative Store Bypass: Vulnerable [ 0.063230] debug: unmapping init [mem 0xffffffffb2e59000-0xffffffffb2e60fff] [ 0.066498] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.068509] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.069022] ... version: 2 [ 0.070025] ... bit width: 48 [ 0.071059] ... generic registers: 4 [ 0.072017] ... value mask: 0000ffffffffffff [ 0.073019] ... max period: 00007fffffffffff [ 0.074021] ... fixed-purpose events: 3 [ 0.075033] ... event mask: 000000070000000f [ 0.076286] rcu: Hierarchical SRCU implementation. [ 0.083146] smp: Bringing up secondary CPUs ... [ 0.084816] x86: Booting SMP configuration: [ 0.085042] .... node #0, CPUs: #1 #2 #3 [ 0.099302] smp: Brought up 1 node, 4 CPUs [ 0.101189] smpboot: Max logical packages: 1 [ 0.102017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239080] node 0 deferred pages initialised in 131ms [ 0.243220] devtmpfs: initialized [ 0.244383] x86/mm: Memory block size: 128MB [ 0.247463] gcov: version magic: 0x41383552 [ 0.250621] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.255353] pinctrl core: initialized pinctrl subsystem [ 0.258274] [ 0.259007] ************************************************************* [ 0.263012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.266011] ** ** [ 0.270011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.274010] ** ** [ 0.277015] ** This means that this kernel is built to expose internal ** [ 0.281014] ** IOMMU data structures, which may compromise security on ** [ 0.283011] ** your system. ** [ 0.286009] ** ** [ 0.289020] ** If you see this message and you are not debugging the ** [ 0.294011] ** kernel, report this immediately to your vendor! ** [ 0.302015] ** ** [ 0.308013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.316014] ************************************************************* [ 0.327582] NET: Registered protocol family 16 [ 0.334779] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.342164] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.354555] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.365083] cpuidle: using governor menu [ 0.372591] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.380374] PCI: Using configuration type 1 for base access [ 0.386625] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.419090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.429018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.439000] cryptd: max_cpu_qlen set to 1000 [ 0.445199] ACPI: Added _OSI(Module Device) [ 0.446015] ACPI: Added _OSI(Processor Device) [ 0.448099] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.451013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.457896] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.474049] ACPI: Interpreter enabled [ 0.478072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.482014] ACPI: Using IOAPIC for interrupt routing [ 0.488105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.500174] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.546000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.552046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.557020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.565907] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.582353] acpiphp: Slot [2] registered [ 0.584407] acpiphp: Slot [5] registered [ 0.586136] acpiphp: Slot [6] registered [ 0.588103] acpiphp: Slot [7] registered [ 0.590113] acpiphp: Slot [8] registered [ 0.591214] acpiphp: Slot [9] registered [ 0.593219] acpiphp: Slot [10] registered [ 0.595311] acpiphp: Slot [3] registered [ 0.597078] acpiphp: Slot [4] registered [ 0.599090] acpiphp: Slot [11] registered [ 0.601098] acpiphp: Slot [12] registered [ 0.602267] acpiphp: Slot [13] registered [ 0.604200] acpiphp: Slot [14] registered [ 0.606091] acpiphp: Slot [15] registered [ 0.607056] acpiphp: Slot [16] registered [ 0.609090] acpiphp: Slot [17] registered [ 0.611105] acpiphp: Slot [18] registered [ 0.613230] acpiphp: Slot [19] registered [ 0.615114] acpiphp: Slot [20] registered [ 0.617082] acpiphp: Slot [21] registered [ 0.619061] acpiphp: Slot [22] registered [ 0.620059] acpiphp: Slot [23] registered [ 0.622153] acpiphp: Slot [24] registered [ 0.624103] acpiphp: Slot [25] registered [ 0.626087] acpiphp: Slot [26] registered [ 0.627170] acpiphp: Slot [27] registered [ 0.629084] acpiphp: Slot [28] registered [ 0.631203] acpiphp: Slot [29] registered [ 0.633168] acpiphp: Slot [30] registered [ 0.634093] acpiphp: Slot [31] registered [ 0.636054] PCI host bridge to bus 0000:00 [ 0.638018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.640017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.643022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.645018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.649020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.651019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.655169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.663177] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.665000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.681012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.689053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.692020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.694069] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.697014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.700670] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.702908] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.706110] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.709861] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.714013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.725007] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.731012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.736000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.818016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.846017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.891061] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.908758] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.926016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.927000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.928031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.942000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.948021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.960016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.997017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.014255] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.025118] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.035015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.057015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.072682] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.083015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.094015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.123018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.143000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.152071] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.166015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.199015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.219836] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.223680] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.226350] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.228507] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.231176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.236159] iommu: Default domain type: Passthrough [ 1.238070] SCSI subsystem initialized [ 1.240128] ACPI: bus type USB registered [ 1.241206] usbcore: registered new interface driver usbfs [ 1.243215] usbcore: registered new interface driver hub [ 1.245213] usbcore: registered new device driver usb [ 1.248156] pps_core: LinuxPPS API ver. 1 registered [ 1.249009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.252110] PTP clock support registered [ 1.255068] EDAC MC: Ver: 3.0.0 [ 1.256000] PCI: Using ACPI for IRQ routing [ 1.256000] NetLabel: Initializing [ 1.257008] NetLabel: domain hash size = 128 [ 1.259009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.261085] NetLabel: unlabeled traffic allowed by default [ 1.263380] vgaarb: loaded [ 1.265643] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.267011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.273539] clocksource: Switched to clocksource kvm-clock [ 1.820251] VFS: Disk quotas dquot_6.6.0 [ 1.825884] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.833918] *** VALIDATE ramfs *** [ 1.838341] *** VALIDATE hugetlbfs *** [ 1.848554] pnp: PnP ACPI init [ 1.854635] pnp: PnP ACPI: found 6 devices [ 1.942043] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.947516] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.960928] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.973808] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.984535] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.993603] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 2.000944] NET: Registered protocol family 2 [ 2.008606] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.036451] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.088117] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.160802] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.197976] TCP: Hash tables configured (established 65536 bind 65536) [ 2.203941] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.214375] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.219926] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.228496] NET: Registered protocol family 1 [ 2.233915] RPC: Registered named UNIX socket transport module. [ 2.236865] RPC: Registered udp transport module. [ 2.239154] RPC: Registered tcp transport module. [ 2.241732] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.246084] NET: Registered protocol family 44 [ 2.248276] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.251702] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.253248] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.254820] PCI: CLS 0 bytes, default 64 [ 2.255881] Unpacking initramfs... [ 6.537435] debug: unmapping init [mem 0xffff9363fcc54000-0xffff9363fffbffff] [ 6.545662] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.551179] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.558612] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.905234] Initialise system trusted keyrings [ 8.907765] Key type blacklist registered [ 8.910196] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.920112] zbud: loaded [ 8.923092] *** VALIDATE nfs *** [ 8.924141] *** VALIDATE nfs4 *** [ 8.926103] pstore: using deflate compression [ 8.930210] Platform Keyring initialized [ 9.086947] NET: Registered protocol family 38 [ 9.094776] Key type asymmetric registered [ 9.099499] Asymmetric key parser 'x509' registered [ 9.103966] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 9.111220] io scheduler mq-deadline registered [ 9.114333] io scheduler kyber registered [ 9.118731] io scheduler bfq registered [ 9.122730] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 9.129579] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 9.132629] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 9.135596] ACPI: Power Button [PWRF] [ 9.145229] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.152614] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.181960] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.191116] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.215967] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.242470] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.273466] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.278274] Non-volatile memory driver v1.3 [ 9.280180] Linux agpgart interface v0.103 [ 9.312601] virtio_blk virtio1: [vda] 134496 512-byte logical blocks (68.9 MB/65.7 MiB) [ 9.314995] vda: detected capacity change from 0 to 68861952 [ 9.357965] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.361714] vdb: detected capacity change from 0 to 1073741824 [ 9.390189] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.394083] vdc: detected capacity change from 0 to 2621440000 [ 9.415910] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.420819] vdd: detected capacity change from 0 to 2621440000 [ 9.441990] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.445854] vde: detected capacity change from 0 to 4294967296 [ 9.465155] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.468211] vdf: detected capacity change from 0 to 4294967296 [ 9.477610] libphy: Fixed MDIO Bus: probed [ 9.502880] usbcore: registered new interface driver usbserial_generic [ 9.507172] usbserial: USB Serial support registered for generic [ 9.513581] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.523691] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.526489] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.534509] mousedev: PS/2 mouse device common for all mice [ 9.545081] rtc_cmos 00:05: RTC can wake from S4 [ 9.562261] rtc_cmos 00:05: registered as rtc0 [ 9.566923] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.569015] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.569059] intel_pstate: CPU model not supported [ 9.576927] hid: raw HID events driver (C) Jiri Kosina [ 9.592947] usbcore: registered new interface driver usbhid [ 9.599179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.604294] usbhid: USB HID core driver [ 9.606450] drop_monitor: Initializing network drop monitor service [ 9.619244] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.624560] Initializing XFRM netlink socket [ 9.637805] NET: Registered protocol family 10 [ 9.645068] Segment Routing with IPv6 [ 9.649575] NET: Registered protocol family 17 [ 9.659046] mpls_gso: MPLS GSO support [ 9.673957] RAS: Correctable Errors collector initialized. [ 9.683723] AVX version of gcm_enc/dec engaged. [ 9.691799] AES CTR mode by8 optimization enabled [ 10.002419] sched_clock: Marking stable (10002399219, 0)->(14104057054, -4101657835) [ 10.026437] registered taskstats version 1 [ 10.033873] Loading compiled-in X.509 certificates [ 10.037663] zswap: loaded using pool lzo/zbud [ 10.090797] Key type big_key registered [ 10.138402] Key type encrypted registered [ 10.144248] ima: No TPM chip found, activating TPM-bypass! [ 10.151693] ima: Allocated hash algorithm: sha1 [ 10.156780] ima: No architecture policies found [ 10.159628] evm: Initialising EVM extended attributes: [ 10.162035] evm: security.selinux [ 10.163883] evm: security.ima [ 10.165901] evm: security.capability [ 10.167702] evm: HMAC attrs: 0x1 [ 10.178157] rtc_cmos 00:05: setting system clock to 2026-04-29 00:40:27 UTC (1777423227) [ 10.186418] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 10.205798] debug: unmapping init [mem 0xffffffffb2b82000-0xffffffffb2e58fff] [ 10.223947] Write protecting the kernel read-only data: 28672k [ 10.247822] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 10.258033] debug: unmapping init [mem 0xffffffffb1b14000-0xffffffffb1bfffff] [ 10.531715] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.558987] systemd[1]: Detected virtualization kvm. [ 10.560442] systemd[1]: Detected architecture x86-64. [ 10.564848] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.648959] systemd[1]: No hostname configured. [ 10.653526] systemd[1]: Set hostname to . [ 10.667826] random: systemd: uninitialized urandom read (16 bytes read) [ 10.678507] systemd[1]: Initializing machine ID from random generator. [ 10.915514] random: ln: uninitialized urandom read (6 bytes read) [ 11.336948] random: systemd: uninitialized urandom read (16 bytes read) [ 11.350750] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 11.373648] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 11.398918] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 13.489988] device-mapper: uevent: version 1.0.3 [ 13.499604] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 15.674073] virtio_net virtio0 ens2: renamed from eth0 [ 15.838166] random: fast init done [ 16.165199] scsi host0: ata_piix [ 16.238132] scsi host1: ata_piix [ 16.247966] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 16.253972] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 20.699286] random: crng init done [ 20.703430] random: 7 urandom warning(s) missed due to ratelimiting [ 23.894665] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 25.548353] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 29.555274] printk: systemd: 26 output lines suppressed due to ratelimiting [ 30.718723] SELinux: Disabled at runtime. [ 30.938136] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 30.962078] systemd[1]: Detected virtualization kvm. [ 30.968526] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.370677] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.382708] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.406264] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.415736] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.425469] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 33.435865] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.453225] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 33.799191] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 35.922197] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.701120] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.788976] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.489919] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.732339] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit)[ 46.663240] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit)[ 48.056611] NFS: Registering the id_resolver key type [ 48.069263] Key type id_resolver registered [ 48.079410] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg249-server login: [ 97.269443] libcfs: loading out-of-tree module taints kernel. [ 97.299915] Key type ._llcrypt registered [ 97.301861] Key type .llcrypt registered [ 97.387118] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 109.118699] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 110.269621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.284354] alg: No test for adler32 (adler32-zlib) [ 111.539618] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 112.134555] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 113.792185] Key type lgssc registered [ 114.971809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.750783] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.075805] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 166.359298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.384659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.566496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.585858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.652662] Lustre: lustre-MDT0000: new disk, initializing [ 167.721814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.738460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.482343] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.634170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.709827] Lustre: 6489:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.734566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.738502] Lustre: Skipped 1 previous similar message [ 179.819746] Lustre: lustre-MDT0001: new disk, initializing [ 179.887291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.919767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.941141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.059782] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.846849] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.852931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.054044] Lustre: lustre-OST0000: new disk, initializing [ 193.057786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.101273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.083455] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.174681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.184640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.240223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.061281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.134278] Lustre: lustre-OST0001: new disk, initializing [ 206.138789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.169349] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.522322] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.620685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.629415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.706066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.506385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.007861] hrtimer: interrupt took 7405788 ns [ 224.959670] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.575903] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.137547] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing yml_node [ 241.210722] Lustre: DEBUG MARKER: Client: 2.17.51.75 [ 243.301757] Lustre: DEBUG MARKER: MDS: 2.17.51.75 [ 245.388113] Lustre: DEBUG MARKER: OSS: 2.17.51.75 [ 246.550472] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 28 20:44:22 EDT 2026 [ 259.621408] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 260.624560] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 276.965448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.967624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 276.973977] Lustre: Skipped 1 previous similar message [ 276.989095] Lustre: Skipped 3 previous similar messages [ 282.081254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.201296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.211513] Lustre: Skipped 6 previous similar messages [ 289.248448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 289.397914] Lustre: server umount lustre-MDT0000 complete [ 292.643169] LustreError: 6479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423509 with bad export cookie 8604973815047311109 [ 292.644286] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.652942] LustreError: 6479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 297.448710] LustreError: 6494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.456982] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.459936] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.470446] LustreError: 6494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 297.488232] Lustre: Skipped 3 previous similar messages [ 299.128664] Lustre: server umount lustre-MDT0001 complete [ 302.346039] Lustre: server umount lustre-OST0000 complete [ 305.763659] Lustre: server umount lustre-OST0001 complete [ 310.970839] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 318.187172] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 346.909739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.271526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 347.320474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 347.460366] Lustre: lustre-MDT0000: new disk, initializing [ 347.587232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.614426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 351.455142] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 359.976182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.065354] Lustre: 14958:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 360.129957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 360.133561] Lustre: Skipped 1 previous similar message [ 360.228451] Lustre: lustre-MDT0001: new disk, initializing [ 360.314615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 360.347052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 360.358399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 364.195708] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 370.525865] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 374.176879] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 379.644159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 379.929909] Lustre: lustre-OST0000: new disk, initializing [ 379.937470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 380.035407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.082622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 381.094588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 381.190630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 384.729471] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 391.382239] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 394.704500] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 394.902296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 397.999889] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 398.131765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 401.889404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 401.897346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.903690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 401.906429] Lustre: Skipped 1 previous similar message [ 401.913051] Lustre: Skipped 1 previous similar message [ 405.713419] Lustre: server umount lustre-OST0000 complete [ 411.622084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 411.631374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.645278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.655094] Lustre: Skipped 1 previous similar message [ 414.913845] Lustre: server umount lustre-MDT0000 complete [ 417.250881] LustreError: 14965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 417.273463] LustreError: 14965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 417.743038] LustreError: 14949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423635 with bad export cookie 8604973815047312642 [ 417.756713] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 417.757296] LustreError: 14949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 417.988113] Lustre: server umount lustre-MDT0001 complete [ 422.895796] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:47:18 (1777423638) [ 439.518771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.875526] LustreError: 19428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 439.942654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.645497] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 445.419560] LustreError: 19429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.531855] LustreError: 19428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.367211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.016994] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 460.859725] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 463.837203] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.720897] Lustre: Failing over lustre-MDT0000 [ 465.923719] Lustre: server umount lustre-MDT0000 complete [ 466.915695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.924733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.937445] Lustre: Skipped 1 previous similar message [ 466.948891] LustreError: 20138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.326765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.402138] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.756892] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 488.929859] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 488.945124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 488.949969] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 492.684659] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 497.125788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.136362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.137653] Lustre: Skipped 2 previous similar messages [ 497.142252] Lustre: Skipped 2 previous similar messages [ 502.340364] Lustre: server umount lustre-MDT0000 complete [ 505.655946] LustreError: 19409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423722 with bad export cookie 8604973815047314056 [ 505.661709] LustreError: 19409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 505.667050] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.850765] Lustre: server umount lustre-MDT0001 complete [ 512.295597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.639237] LustreError: 22420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.657172] LustreError: 22420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 512.700357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.706882] Lustre: Skipped 2 previous similar messages [ 515.590859] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 522.861721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.886325] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 531.963846] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 535.277828] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 537.130432] Lustre: Failing over lustre-MDT0000 [ 537.309060] Lustre: server umount lustre-MDT0000 complete [ 538.593888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.608379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.624299] Lustre: Skipped 1 previous similar message [ 547.815194] LustreError: 22417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.833535] LustreError: 22417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 553.954373] Lustre: 3639:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777423755/real 1777423755] req@ffff936448516300 x1863763444800256/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777423771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 553.999811] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.455142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.195207] LustreError: 3638:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff936448516300 x1863763444802688/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 567.973402] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 568.547971] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 568.553709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 568.568527] Lustre: Skipped 1 previous similar message [ 568.582845] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 574.021424] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 579.041899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.043700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.046102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.059260] Lustre: Skipped 1 previous similar message [ 579.073417] Lustre: Skipped 4 previous similar messages [ 583.719763] Lustre: server umount lustre-MDT0000 complete [ 587.110680] LustreError: 22402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423804 with bad export cookie 8604973815047314994 [ 587.117322] LustreError: 22402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 587.117898] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.381588] Lustre: server umount lustre-MDT0001 complete [ 594.567660] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:50:10 (1777423810) [ 606.238802] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 613.445118] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 643.145143] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 650.709684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.912157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 650.934756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 650.993956] Lustre: lustre-MDT0000: new disk, initializing [ 651.057893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.061981] Lustre: Skipped 2 previous similar messages [ 651.091591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 654.029233] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 661.333747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.417472] Lustre: 28395:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 661.625453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 664.540824] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 669.424255] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 671.611804] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 676.253328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.208188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 678.237244] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 678.282461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 678.294406] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 680.823394] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 686.267783] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 691.532765] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 703.982959] Lustre: server umount lustre-OST0000 complete [ 707.558555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 707.566216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.575053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.306798] Lustre: server umount lustre-MDT0000 complete [ 718.340398] LustreError: 29185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423935 with bad export cookie 8604973815047315400 [ 718.343970] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.351846] LustreError: 29185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 728.694082] Lustre: server umount lustre-MDT0001 complete [ 735.012367] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 736.809185] Key type lgssc unregistered [ 737.048180] LNet: 31964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.056254] LNetError: 31964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.073584] LNet: Removed LNI 192.168.202.149@tcp [ 737.625166] Key type .llcrypt unregistered [ 737.627213] Key type ._llcrypt unregistered [ 753.449997] Key type ._llcrypt registered [ 753.452303] Key type .llcrypt registered [ 753.530141] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 754.165794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.269335] alg: No test for adler32 (adler32-zlib) [ 755.222643] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 755.376826] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 756.993312] Key type lgssc registered [ 757.654538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 762.410670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 762.437496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.790827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.541254] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 772.186873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.434676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.408455] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 779.471281] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 781.855657] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 786.153687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 786.371169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.793447] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 793.117284] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 798.708416] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 806.422881] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:53:42 (1777424022) [ 807.492629] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 808.535875] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 812.873218] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:53:48 (1777424028) [ 818.613885] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:53:54 (1777424034) [ 819.374230] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 825.399940] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 825.403918] Lustre: Skipped 2 previous similar messages [ 826.513349] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:54:02 (1777424042) [ 831.357985] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 831.363911] Lustre: Skipped 4 previous similar messages [ 839.748159] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 839.753744] Lustre: Skipped 8 previous similar messages [ 840.798696] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:54:16 (1777424056) [ 842.327265] Lustre: *** cfs_fail_loc=131b, val=2*** [ 844.363552] Lustre: *** cfs_fail_loc=131b, val=2*** [ 849.271414] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:54:25 (1777424065) [ 857.476750] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 857.481308] Lustre: Skipped 4 previous similar messages [ 858.534244] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:54:34 (1777424074) [ 866.275280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.279053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.285740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 870.371723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 870.373959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 870.375992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.191590] Lustre: server umount lustre-OST0000 complete [ 874.978971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 874.991297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.000231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 875.003059] Lustre: Skipped 1 previous similar message [ 880.611200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 880.617051] Lustre: Skipped 3 previous similar messages [ 880.747489] Lustre: server umount lustre-MDT0000 complete [ 882.979840] LustreError: 35920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424100 with bad export cookie 4661831478449020099 [ 882.987460] LustreError: 35920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 882.987550] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 883.185564] Lustre: server umount lustre-MDT0001 complete [ 889.905588] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 891.661801] Key type lgssc unregistered [ 891.886791] LNet: 40830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.896695] LNetError: 40830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.912908] LNet: Removed LNI 192.168.202.149@tcp [ 892.504792] Key type .llcrypt unregistered [ 892.506802] Key type ._llcrypt unregistered [ 903.980666] Key type ._llcrypt registered [ 903.982435] Key type .llcrypt registered [ 904.043571] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 914.131354] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 914.663788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.733242] alg: No test for adler32 (adler32-zlib) [ 915.692600] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 915.838799] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 917.464256] Key type lgssc registered [ 918.249788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.164561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 939.177691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.330537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 940.352582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 940.403877] Lustre: lustre-MDT0000: new disk, initializing [ 940.460213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.480610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 943.019706] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 949.290862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.391458] Lustre: 43926:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 949.443088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 949.446533] Lustre: Skipped 1 previous similar message [ 949.494912] Lustre: lustre-MDT0001: new disk, initializing [ 949.549987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 949.585279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 949.592881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 951.701151] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 956.182788] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 958.649847] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 962.721743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 962.862439] Lustre: lustre-OST0000: new disk, initializing [ 962.866092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 962.923316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.507336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 964.515064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 964.591751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 966.298978] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 971.230460] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.219835] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 973.329799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 975.404267] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 975.516541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 979.939070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.943277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 979.944500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.946591] Lustre: Skipped 1 previous similar message [ 979.962075] LustreError: Skipped 1 previous similar message [ 982.669745] Lustre: server umount lustre-OST0000 complete [ 985.058519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.059579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.068685] Lustre: Skipped 1 previous similar message [ 985.076343] Lustre: Skipped 2 previous similar messages [ 990.179267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.190544] Lustre: Skipped 1 previous similar message [ 995.305251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.310141] Lustre: Skipped 1 previous similar message [ 999.392124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 999.514689] Lustre: server umount lustre-MDT0000 complete [ 1002.020124] LustreError: 47058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424219 with bad export cookie 13033046966954938467 [ 1002.022404] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.029073] LustreError: 47058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1002.205297] Lustre: server umount lustre-MDT0001 complete [ 1008.068105] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:57:04 (1777424224) [ 1012.632398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.921212] LustreError: 47980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.936950] LustreError: 47980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1012.983304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.465373] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1018.337653] LustreError: 47981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.406117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.623297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1022.952882] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1026.394072] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1028.146581] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1031.666141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1031.881017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.953955] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1037.371963] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1049.942454] LustreError: 50192:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1050.396928] Lustre: Modifying parameter general.timeout=1 in log params [ 1051.702117] Lustre: Modifying parameter general.timeout=4 in log params [ 1051.704062] Lustre: Skipped 2 previous similar messages [ 1053.904955] Lustre: Modifying parameter general.timeout=8 in log params [ 1053.907101] Lustre: Skipped 3 previous similar messages [ 1058.179086] Lustre: Modifying parameter general.timeout=17 in log params [ 1058.181170] Lustre: Skipped 8 previous similar messages [ 1066.416655] Lustre: Modifying parameter general.timeout=34 in log params [ 1066.418974] Lustre: Skipped 16 previous similar messages [ 1082.756808] Lustre: Modifying parameter general.timeout=64 in log params [ 1082.758975] Lustre: Skipped 29 previous similar messages [ 1115.169485] Lustre: Modifying parameter general.timeout=127 in log params [ 1115.172651] Lustre: Skipped 62 previous similar messages [ 1122.382925] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:58:58 (1777424338) [ 1123.942732] Lustre: 57110:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1124.951856] Lustre: 57208:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1127.112485] Lustre: 57406:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1127.120578] Lustre: 57406:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1143.592384] Lustre: 58953:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1143.596104] Lustre: 58953:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1143.608708] LustreError: 58953:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1145.910605] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:59:22 (1777424362) [ 1152.486434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.487843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.494772] Lustre: Skipped 2 previous similar messages [ 1152.497860] Lustre: Skipped 1 previous similar message [ 1158.084189] Lustre: server umount lustre-MDT0000 complete [ 1159.619141] LustreError: 50244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424376 with bad export cookie 13033046966954939349 [ 1159.621592] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.624805] LustreError: 50244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1159.811761] Lustre: server umount lustre-MDT0001 complete [ 1172.078143] Lustre: server umount lustre-OST0000 complete [ 1183.983693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.142183] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1184.156290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1184.164099] Lustre: Skipped 25 previous similar messages [ 1184.289839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.130809] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1190.438763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.514756] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1190.536533] Lustre: 61477:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1193.031921] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1195.909498] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1197.699839] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1201.202405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1201.310571] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1204.505018] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1207.250101] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1208.142758] Lustre: 63702:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1212.388099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.398753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.401102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1212.411280] Lustre: Skipped 1 previous similar message [ 1212.414294] Lustre: Skipped 4 previous similar messages [ 1213.315883] Lustre: server umount lustre-OST0000 complete [ 1216.486479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1216.490225] LustreError: Skipped 1 previous similar message [ 1216.492158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.636294] Lustre: server umount lustre-MDT0000 complete [ 1222.625854] LustreError: 61486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.645759] LustreError: 61486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1223.849600] LustreError: 61515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424441 with bad export cookie 13033046966954941904 [ 1223.850762] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.859213] LustreError: 61515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1224.061741] Lustre: server umount lustre-MDT0001 complete [ 1230.091776] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1231.693790] Key type lgssc unregistered [ 1231.851787] LNet: 64892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.860198] LNetError: 64892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.875621] LNet: Removed LNI 192.168.202.149@tcp [ 1232.333883] Key type .llcrypt unregistered [ 1232.337442] Key type ._llcrypt unregistered [ 1246.275655] Key type ._llcrypt registered [ 1246.278799] Key type .llcrypt registered [ 1246.362790] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1247.023390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.047906] alg: No test for adler32 (adler32-zlib) [ 1247.989269] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1248.159524] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1249.808169] Key type lgssc registered [ 1250.420854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.222909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1255.237394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.514658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.591170] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1263.390566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.719385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.595148] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1268.155891] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1269.773680] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1272.732427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.002535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.663705] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1278.257654] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1287.676521] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1291.748435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.749593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1291.753456] Lustre: Skipped 2 previous similar messages [ 1291.758035] Lustre: Skipped 2 previous similar messages [ 1296.866255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.868875] Lustre: Skipped 2 previous similar messages [ 1297.137172] Lustre: server umount lustre-MDT0000 complete [ 1298.912183] LustreError: 66258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424516 with bad export cookie 3205872095950607794 [ 1298.914712] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1298.920236] LustreError: 66258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1299.080379] Lustre: server umount lustre-MDT0001 complete [ 1311.294436] Lustre: server umount lustre-OST0000 complete [ 1324.136199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1324.327816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1324.351138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1324.535078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.829888] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1331.311801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.390129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1331.405601] Lustre: 70816:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1331.449957] LustreError: 70826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.458099] LustreError: 70826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1331.633786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1333.935500] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1337.062235] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1339.157058] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1342.992886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1343.107067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1343.332220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.303278] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1349.267730] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1350.120339] Lustre: 73050:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1351.084103] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1358.306664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1358.307696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.310045] LustreError: Skipped 1 previous similar message [ 1358.310294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1358.320722] Lustre: Skipped 1 previous similar message [ 1359.662856] Lustre: server umount lustre-OST0000 complete [ 1362.401820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1362.407735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.418407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.420918] Lustre: Skipped 1 previous similar message [ 1368.544988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1368.548386] Lustre: Skipped 3 previous similar messages [ 1376.224137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1376.301333] Lustre: server umount lustre-MDT0000 complete [ 1377.762300] LustreError: 70826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.770188] LustreError: 70826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1378.094762] LustreError: 70808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424595 with bad export cookie 3205872095950608627 [ 1378.096824] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.100741] LustreError: 70808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1378.240506] Lustre: server umount lustre-MDT0001 complete [ 1382.880432] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1384.129440] Key type lgssc unregistered [ 1384.270569] LNet: 74242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.274372] LNetError: 74242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.290595] LNet: Removed LNI 192.168.202.149@tcp [ 1384.619707] Key type .llcrypt unregistered [ 1384.621744] Key type ._llcrypt unregistered [ 1396.325651] Key type ._llcrypt registered [ 1396.327195] Key type .llcrypt registered [ 1396.369796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1396.936764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.960411] alg: No test for adler32 (adler32-zlib) [ 1397.858850] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1397.958294] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1399.552190] Key type lgssc registered [ 1400.056371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1403.841460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1403.849543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.074314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.867655] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1410.771178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.077061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.025426] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1415.485161] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.070811] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1420.096279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.404489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.042621] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1425.783743] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1434.179396] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1434.983848] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1434.986792] Lustre: Skipped 1 previous similar message [ 1436.212727] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1436.215579] Lustre: Skipped 2 previous similar messages [ 1438.521712] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1438.525102] Lustre: Skipped 5 previous similar messages [ 1442.955366] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1442.959710] Lustre: Skipped 9 previous similar messages [ 1451.298767] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1451.304146] Lustre: Skipped 17 previous similar messages [ 1467.737427] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1467.741652] Lustre: Skipped 39 previous similar messages [ 1481.262705] Lustre: 83350:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1482.478180] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:04:58 (1777424698) [ 1483.147384] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1483.934620] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:05:00 (1777424700) [ 1487.333569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.334895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.339884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1487.344733] Lustre: Skipped 1 previous similar message [ 1491.269057] Lustre: server umount lustre-OST0000 complete [ 1497.570937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.573768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1497.582175] Lustre: Skipped 2 previous similar messages [ 1499.212949] Lustre: server umount lustre-MDT0000 complete [ 1501.036728] LustreError: 75594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424718 with bad export cookie 1612871853305256826 [ 1501.041493] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.047526] LustreError: 75594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1501.250091] Lustre: server umount lustre-MDT0001 complete [ 1506.559790] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1507.885405] Key type lgssc unregistered [ 1508.033436] LNet: 84578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.037579] LNetError: 84578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.045774] LNet: Removed LNI 192.168.202.149@tcp [ 1508.441308] Key type .llcrypt unregistered [ 1508.442603] Key type ._llcrypt unregistered [ 1510.363964] Key type ._llcrypt registered [ 1510.365346] Key type .llcrypt registered [ 1510.411068] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1512.756843] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules [ 1513.374538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.391413] alg: No test for adler32 (adler32-zlib) [ 1514.286571] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1514.434821] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1516.040126] Key type lgssc registered [ 1516.693863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1517.227859] LustreError: 85706:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1525.746531] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1529.604806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1538.164123] LustreError: 85706:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1538.173585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.397358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.979920] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1544.791913] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:06:01 (1777424761) [ 1546.952478] LustreError: 86744:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1546.956821] LustreError: 86744:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1546.960395] LustreError: 86744:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1547.093340] Lustre: server umount lustre-MDT0000 complete [ 1552.395970] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1553.580136] Key type lgssc unregistered [ 1553.720408] LNet: 88201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.726772] LNetError: 88201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.735911] LNet: Removed LNI 192.168.202.149@tcp [ 1554.102879] Key type .llcrypt unregistered [ 1554.104972] Key type ._llcrypt unregistered [ 1562.158352] Key type ._llcrypt registered [ 1562.160093] Key type .llcrypt registered [ 1562.212662] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1562.596216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.633425] alg: No test for adler32 (adler32-zlib) [ 1563.525527] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1563.619044] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1565.209946] Key type lgssc registered [ 1565.712461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1569.265871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1569.276704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.499399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.976858] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1575.359815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.628626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1577.169578] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1579.360830] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1580.741662] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1583.285163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1583.563187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.733758] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1588.050582] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1623.573138] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:07:19 (1777424839) [ 1624.581229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1625.498932] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:07:21 (1777424841) [ 1630.691388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.693387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.699293] Lustre: Skipped 2 previous similar messages [ 1630.702592] Lustre: Skipped 1 previous similar message [ 1632.933300] Lustre: server umount lustre-MDT0000 complete [ 1634.816656] LustreError: 90104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424852 with bad export cookie 1081094898729699004 [ 1634.819977] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1634.823152] LustreError: 90104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1634.961238] Lustre: server umount lustre-MDT0001 complete [ 1647.114892] Lustre: server umount lustre-OST0000 complete [ 1651.854706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.068497] LustreError: 93335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.078891] LustreError: 93335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1652.179381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.875801] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1657.187334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.452463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1658.996795] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1661.107142] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1662.638501] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1667.374458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1667.454865] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1667.458840] LustreError: 93323:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1667.468508] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1667.472351] LustreError: 95190:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1667.475275] LustreError: 95190:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1667.477669] LustreError: 95190:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1667.503126] Lustre: server umount lustre-OST0000 complete [ 1667.504793] LustreError: 95190:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1670.713084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1670.785217] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1670.789137] LustreError: 93325:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1670.799448] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1670.807234] LustreError: 95657:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1670.810524] LustreError: 95657:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1670.813202] LustreError: 95657:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1670.839940] Lustre: server umount lustre-OST0000 complete [ 1670.841972] LustreError: 95657:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1674.690722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1674.769402] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1674.771764] Lustre: Found index 0 for lustre-OST0000, updating log [ 1674.775434] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1674.810297] Lustre: lustre-OST0000: new disk, initializing [ 1674.813055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1674.938244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.140709] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1678.587954] Lustre: Failing over lustre-OST0000 [ 1678.646246] Lustre: server umount lustre-OST0000 complete [ 1682.401641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.402247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.406091] Lustre: Skipped 1 previous similar message [ 1682.408366] Lustre: Skipped 1 previous similar message [ 1683.427657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.431870] Lustre: Skipped 2 previous similar messages [ 1688.545247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.547624] Lustre: Skipped 1 previous similar message [ 1690.620242] Lustre: server umount lustre-MDT0000 complete [ 1691.976152] LustreError: 93317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424909 with bad export cookie 1081094898729701748 [ 1691.978722] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1691.981619] LustreError: 93317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.095071] Lustre: server umount lustre-MDT0001 complete [ 1695.176545] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:08:31 (1777424911) [ 1700.378133] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1704.169048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.373258] LustreError: 98495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.404897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.008648] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1709.063459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.775486] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1711.870599] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1714.308675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1714.568778] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1716.490555] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1718.691513] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1718.695600] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:09 [ 1718.701248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1718.702295] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1718.704082] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1718.706201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1719.380317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.410272] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1719.441029] Lustre: lustre-OST0001: new disk, initializing [ 1719.442834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1719.808213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1719.812507] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1719.843834] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1721.459068] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1724.580932] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1729.207353] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1737.775429] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.49@tcp (stopping) [ 1740.259569] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.260212] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1740.265483] Lustre: Skipped 1 previous similar message [ 1741.782316] Lustre: server umount lustre-MDT0001 complete [ 1742.887624] LustreError: 98522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1745.040606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.078436] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1745.080990] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1745.098113] Lustre: 98483:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1745.355823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1745.358904] Lustre: Skipped 3 previous similar messages [ 1746.406711] LustreError: 102427:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1746.410413] LustreError: 102427:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1746.414569] LustreError: 98487:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1746.446335] LustreError: 102436:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.149@tcp: cfg command failed: rc = -17 [ 1746.451447] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1746.451447] [ 1746.773806] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1750.400724] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 21:09:26 (1777424966) [ 1750.502524] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1755.923175] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1762.224750] Lustre: 104325:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1770.442834] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1771.674034] Lustre: 105460:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1827.297989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1827.298401] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.299298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.299303] Lustre: Skipped 3 previous similar messages [ 1827.311521] Lustre: Skipped 3 previous similar messages [ 1830.190381] Lustre: server umount lustre-MDT0000 complete [ 1831.885367] LustreError: 99602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425049 with bad export cookie 1081094898729702693 [ 1831.887869] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1831.891520] LustreError: 99602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1832.073358] Lustre: server umount lustre-MDT0001 complete [ 1842.830048] Lustre: server umount lustre-OST0000 complete [ 1853.912392] Lustre: server umount lustre-OST0001 complete [ 1856.394977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1860.395913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1863.855662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1867.776661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1874.572059] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1878.522577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.528745] Lustre: lustre-MDT0000: reset Object Index mappings [ 1878.716890] LustreError: 109365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1878.752870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.251420] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1883.201541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.210373] Lustre: lustre-MDT0001: reset Object Index mappings [ 1884.845045] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1886.035081] Lustre: 110478:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1888.651088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.955243] LustreError: 110833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.960654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 1891.023599] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1893.043075] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1894.328370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.795685] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 1895.801648] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1896.775668] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1900.035962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1901.391935] Lustre: 112328:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1923.284899] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:12:19 (1777425139) [ 1954.273938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.277030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.277281] Lustre: Skipped 3 previous similar messages [ 1954.279353] Lustre: Skipped 4 previous similar messages [ 1958.314973] Lustre: server umount lustre-MDT0000 complete [ 1959.394533] LustreError: 109366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.404700] LustreError: 109366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1959.697506] LustreError: 109345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425177 with bad export cookie 1081094898729749656 [ 1959.700869] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1959.702102] LustreError: 109345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1964.513617] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.517402] Lustre: Skipped 2 previous similar messages [ 1965.901265] Lustre: server umount lustre-MDT0001 complete [ 1973.512163] Lustre: server umount lustre-OST0000 complete [ 1989.088136] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1989.182792] Lustre: server umount lustre-OST0001 complete [ 1991.513470] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 1994.881145] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2007.278383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.368330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2007.380899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2007.415902] Lustre: lustre-MDT0000: new disk, initializing [ 2007.438646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.441238] Lustre: Skipped 3 previous similar messages [ 2007.446643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2008.836595] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2011.880533] Lustre: server umount lustre-MDT0000 complete [ 2015.056462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.154120] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2015.159592] Lustre: 116538:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2015.168985] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2016.536477] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2018.274560] Lustre: server umount lustre-MDT0000 complete [ 2021.274041] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:13:57 (1777425237) [ 2021.825074] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2022.464215] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:13:58 (1777425238) [ 2028.354701] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2032.098056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.787016] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2036.891397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.935989] Lustre: 118552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2036.964691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2036.967055] Lustre: Skipped 1 previous similar message [ 2037.003686] Lustre: lustre-MDT0001: new disk, initializing [ 2037.054320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2037.058361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2038.482444] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2040.896603] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2043.209287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2043.324701] Lustre: lustre-OST0000: new disk, initializing [ 2043.326965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2045.179432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2045.183234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2045.217847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2045.395428] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2049.845509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2051.465465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2051.469381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2051.503617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2052.058388] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2056.644463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2057.848663] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2224.852267] Lustre: 127314:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2224.854902] Lustre: 127314:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2224.856719] Lustre: 127314:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2224.858787] Lustre: 127314:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2224.861699] Lustre: 127314:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2224.863432] Lustre: 127314:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2297.617683] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:18:33 (1777425513) [ 2312.678485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.680571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2312.682770] Lustre: Skipped 3 previous similar messages [ 2312.684771] Lustre: Skipped 4 previous similar messages [ 2315.259755] Lustre: server umount lustre-MDT0000 complete [ 2316.594227] LustreError: 119273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425533 with bad export cookie 1081094898729804445 [ 2316.598758] LustreError: 119273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2316.600167] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.793416] LustreError: 121726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.802089] LustreError: 121726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2318.799810] Lustre: server umount lustre-MDT0001 complete [ 2322.241695] Lustre: server umount lustre-OST0000 complete [ 2329.701680] Lustre: server umount lustre-OST0001 complete [ 2331.870784] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 2335.052367] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2348.292134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.377721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2348.388097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2348.389849] Lustre: Skipped 1 previous similar message [ 2348.423896] Lustre: lustre-MDT0000: new disk, initializing [ 2348.425732] Lustre: Skipped 1 previous similar message [ 2348.454119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.456407] Lustre: Skipped 5 previous similar messages [ 2348.463064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2349.827480] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2354.193646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.230909] Lustre: 130299:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2354.325766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2355.638762] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2358.653452] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2359.788503] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2362.104038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2364.026863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2364.059451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2364.164963] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2367.218436] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2370.395727] Lustre: lustre-MDD0000: changelog on [ 2371.344723] Lustre: lustre-MDD0001: changelog on [ 2742.617367] Lustre: 131804:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2770.339457] Lustre: lustre-MDD0001: changelog off [ 2771.522104] Lustre: lustre-MDD0000: changelog off [ 2773.986928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2773.987241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.989662] LustreError: Skipped 1 previous similar message [ 2773.990223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2773.993371] Lustre: Skipped 3 previous similar messages [ 2773.994525] Lustre: Skipped 2 previous similar messages [ 2779.217627] Lustre: server umount lustre-OST0000 complete [ 2786.479782] Lustre: server umount lustre-MDT0000 complete [ 2787.622115] LustreError: 131079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426004 with bad export cookie 1081094898730568299 [ 2787.624874] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.625489] LustreError: 131079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2787.730978] Lustre: server umount lustre-MDT0001 complete [ 2790.954964] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2791.999330] Key type lgssc unregistered [ 2792.132342] LNet: 140771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2792.135691] LNetError: 140771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.143277] LNet: Removed LNI 192.168.202.149@tcp [ 2792.474123] Key type .llcrypt unregistered [ 2792.475465] Key type ._llcrypt unregistered [ 2798.975149] Key type ._llcrypt registered [ 2798.976965] Key type .llcrypt registered [ 2799.012549] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 2805.117760] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2805.416918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.441044] alg: No test for adler32 (adler32-zlib) [ 2806.304176] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2806.390546] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2807.976180] Key type lgssc registered [ 2808.328192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2819.711206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2819.716337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.802657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2820.815928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2820.852469] Lustre: lustre-MDT0000: new disk, initializing [ 2820.875205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.880670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2822.174378] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2826.214191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.252117] Lustre: 143919:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2826.274198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2826.276022] Lustre: Skipped 1 previous similar message [ 2826.311924] Lustre: lustre-MDT0001: new disk, initializing [ 2826.341412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.351896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2826.355263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2827.627209] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2830.758503] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2831.890171] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2834.115899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.211794] Lustre: lustre-OST0000: new disk, initializing [ 2834.213506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2834.241121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.130671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2835.133696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2835.167827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2836.067403] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2839.113455] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2841.292713] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module obdecho/obdecho [ 2841.906721] LustreError: 146535:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2841.910176] LustreError: 146535:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 2842.522147] LustreError: 146632:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2845.666578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.667309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.669970] Lustre: Skipped 2 previous similar messages [ 2850.786549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.790097] Lustre: Skipped 4 previous similar messages [ 2851.798731] Lustre: server umount lustre-MDT0000 complete [ 2852.942080] LustreError: 143913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426070 with bad export cookie 3852616283870748692 [ 2852.944823] LustreError: 143913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2852.944928] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.044691] Lustre: server umount lustre-MDT0001 complete [ 2864.477286] Lustre: server umount lustre-OST0000 complete [ 2871.498501] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 2875.025023] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2886.886870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.978603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2886.988793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2887.024316] Lustre: lustre-MDT0000: new disk, initializing [ 2887.048485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.054446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2888.235638] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2892.360781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.402275] Lustre: 150304:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2892.424283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2892.426161] Lustre: Skipped 1 previous similar message [ 2892.466463] Lustre: lustre-MDT0001: new disk, initializing [ 2892.505164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2892.517249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2892.520796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2893.834440] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2896.928520] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2898.181887] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2900.447968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.552352] Lustre: lustre-OST0000: new disk, initializing [ 2900.554995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2900.583635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.496982] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2904.187105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2904.189787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2904.231464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2905.594477] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2906.974172] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2907.050963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2908.366295] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2908.437326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2909.666662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.666942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.668817] LustreError: Skipped 1 previous similar message [ 2909.669366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.671715] Lustre: Skipped 1 previous similar message [ 2909.675822] Lustre: Skipped 1 previous similar message [ 2911.122863] Lustre: server umount lustre-OST0000 complete [ 2913.249080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2913.251767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.256028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.555360] Lustre: server umount lustre-MDT0000 complete [ 2919.814874] LustreError: 151081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426137 with bad export cookie 3852616283870749637 [ 2919.818037] LustreError: 151081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2919.819285] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.951622] Lustre: server umount lustre-MDT0001 complete [ 2921.109759] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:28:57 (1777426137) [ 2921.671481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2922.249176] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:28:58 (1777426138) [ 2924.877638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.058666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.327608] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2929.152490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.567161] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2932.415147] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2933.618741] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2935.823877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.805192] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2939.603117] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2960.352805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.355414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.359133] Lustre: Skipped 1 previous similar message [ 2960.360951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.362767] Lustre: Skipped 2 previous similar messages [ 2963.361246] Lustre: server umount lustre-MDT0000 complete [ 2964.625146] LustreError: 154290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426181 with bad export cookie 3852616283870750505 [ 2964.629118] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.629533] LustreError: 154290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2965.542772] LustreError: 156408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.552404] LustreError: 156408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2967.008954] LustreError: 154304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.009165] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.019681] Lustre: Skipped 2 previous similar messages [ 2969.057052] LustreError: 154305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.057325] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2969.065053] Lustre: Skipped 6 previous similar messages [ 2970.809202] Lustre: server umount lustre-MDT0001 complete [ 2972.983408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2977.243783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.333767] LustreError: 157831:0:(llog.c:1646:llog_backup()) MGC192.168.202.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2977.338862] Lustre: 157831:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2994.656867] LustreError: 157835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.661110] LustreError: 157835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2998.752951] LustreError: 157836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.757495] LustreError: 157836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3007.527361] LustreError: 157837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.533704] LustreError: 157837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3026.983223] LustreError: 157835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.992360] LustreError: 157835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 3038.304346] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3038.307857] Lustre: 157831:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3059.169937] LustreError: 157853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.174420] LustreError: 157853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3079.145063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3079.147357] Lustre: Skipped 2 previous similar messages [ 3080.463913] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3083.305914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.260287] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3084.262472] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x35773e1920008051 [ 3085.641641] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3087.428920] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3088.021391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3089.203899] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3089.382020] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3089.383975] Lustre: Skipped 1 previous similar message [ 3093.837501] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 3099.618739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3099.622053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.623404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.627073] Lustre: Skipped 2 previous similar messages [ 3099.629481] Lustre: Skipped 1 previous similar message [ 3100.506497] Lustre: server umount lustre-MDT0000 complete [ 3101.723455] LustreError: 154291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426319 with bad export cookie 3852616283870756945 [ 3101.729747] LustreError: 154291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3101.730659] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3101.843529] Lustre: server umount lustre-MDT0001 complete [ 3113.377292] Lustre: server umount lustre-OST0000 complete [ 3120.413221] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3123.899314] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3135.571338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.661117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3135.670782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3135.703764] Lustre: lustre-MDT0000: new disk, initializing [ 3135.726405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3136.880030] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3141.082702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.126528] Lustre: 162649:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3141.255734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3142.620920] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3145.809160] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3147.042519] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3149.194915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.320057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3149.322237] Lustre: Skipped 3 previous similar messages [ 3151.193767] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3154.259142] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3154.934775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3154.967858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3155.638685] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3155.711418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3157.085725] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3157.158231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3160.035024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3160.035169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.038043] LustreError: Skipped 1 previous similar message [ 3160.038608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.044250] Lustre: Skipped 1 previous similar message [ 3160.045440] Lustre: Skipped 2 previous similar messages [ 3163.922711] Lustre: server umount lustre-OST0000 complete [ 3171.290909] Lustre: server umount lustre-MDT0000 complete [ 3172.498974] LustreError: 163425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426389 with bad export cookie 3852616283870773745 [ 3172.502176] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.503249] LustreError: 163425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3172.620639] Lustre: server umount lustre-MDT0001 complete [ 3175.345105] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:33:11 (1777426391) [ 3177.528216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.683352] LustreError: 166650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.688596] LustreError: 166650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 3178.970922] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3181.818277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.282158] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3185.056183] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3186.275381] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3188.509205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.466290] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3192.234580] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3211.594125] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:33:48 (1777426428) [ 3213.283480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3213.283849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.289687] Lustre: Skipped 3 previous similar messages [ 3218.704943] Lustre: server umount lustre-OST0000 complete [ 3228.641022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.643696] Lustre: Skipped 12 previous similar messages [ 3234.272131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3234.332150] Lustre: server umount lustre-MDT0000 complete [ 3235.601138] LustreError: 166632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426452 with bad export cookie 3852616283870774627 [ 3235.604097] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3235.605041] LustreError: 166632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3235.722478] Lustre: server umount lustre-MDT0001 complete [ 3239.274158] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 3240.288624] Key type lgssc unregistered [ 3240.424398] LNet: 170044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3240.427627] LNetError: 170044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3240.433236] LNet: Removed LNI 192.168.202.149@tcp [ 3240.733540] Key type .llcrypt unregistered [ 3240.734550] Key type ._llcrypt unregistered [ 3248.090204] Key type ._llcrypt registered [ 3248.091289] Key type .llcrypt registered [ 3248.128184] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3248.474490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3248.529337] alg: No test for adler32 (adler32-zlib) [ 3249.385502] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 3249.472910] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 3251.056131] Key type lgssc registered [ 3251.417151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3253.941120] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3253.947655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3269.408405] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3274.528249] LustreError: 171249:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.149@tcp: failed processing log, type 4: rc = -110 [ 3279.648457] LustreError: 171249:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3279.652053] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3279.653728] LustreError: MGC192.168.202.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3279.656744] LustreError: 171249:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3279.659479] LustreError: 171249:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 3279.662465] LustreError: 171249:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 3279.687867] Lustre: server umount lustre-OST0000 complete [ 3279.689292] LustreError: 171249:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3282.554221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.712330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.058828] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3287.966381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.100345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.445542] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3291.215505] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3292.391622] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3294.655536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3294.723188] LustreError: 173337:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3294.728095] Lustre: 173337:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.202.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3294.770651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.617212] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3298.397894] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3302.882427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.884419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.887349] Lustre: Skipped 2 previous similar messages [ 3302.890584] Lustre: Skipped 2 previous similar messages [ 3305.237324] Lustre: server umount lustre-MDT0000 complete [ 3306.460311] LustreError: 171732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426523 with bad export cookie 18165641394863292673 [ 3306.465058] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3306.465373] LustreError: 171732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3306.579877] Lustre: server umount lustre-MDT0001 complete [ 3318.113632] Lustre: server umount lustre-OST0000 complete [ 3321.128792] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3324.182922] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3335.604439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.697360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3335.709612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3335.748720] Lustre: lustre-MDT0000: new disk, initializing [ 3335.771536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.777206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3337.072488] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3341.432745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.470597] Lustre: 176711:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3341.491338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3341.493062] Lustre: Skipped 1 previous similar message [ 3341.528422] Lustre: lustre-MDT0001: new disk, initializing [ 3341.557471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3341.568812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3341.571547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3342.794321] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3345.878820] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3347.116426] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3349.462149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3349.637236] Lustre: lustre-OST0000: new disk, initializing [ 3349.640608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3349.678771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.780531] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3354.886208] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3356.310852] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3359.739409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3359.743138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3359.774429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3360.459674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3361.816717] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3361.891068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3364.835911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3364.836040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.839523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.842642] Lustre: Skipped 1 previous similar message [ 3364.845818] Lustre: Skipped 1 previous similar message [ 3368.661765] Lustre: server umount lustre-OST0000 complete [ 3369.953324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.953964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3369.957412] Lustre: Skipped 1 previous similar message [ 3375.073240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.076141] Lustre: Skipped 2 previous similar messages [ 3380.193847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3380.196371] Lustre: Skipped 1 previous similar message [ 3384.288125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3384.354662] Lustre: server umount lustre-MDT0000 complete [ 3385.313223] LustreError: 176722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.318504] LustreError: 176722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3385.646778] LustreError: 177488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426602 with bad export cookie 18165641394863293359 [ 3385.649660] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3385.651429] LustreError: 177488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3385.784912] Lustre: server umount lustre-MDT0001 complete [ 3388.766727] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:36:45 (1777426605) [ 3394.478942] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 3395.605616] Key type lgssc unregistered [ 3395.738604] LNet: 181255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3395.742371] LNetError: 181255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3395.750504] LNet: Removed LNI 192.168.202.149@tcp [ 3396.060692] Key type .llcrypt unregistered [ 3396.061802] Key type ._llcrypt unregistered [ 3403.051647] Key type ._llcrypt registered [ 3403.052729] Key type .llcrypt registered [ 3403.096790] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3403.483400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3403.523034] alg: No test for adler32 (adler32-zlib) [ 3404.378121] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 3404.467484] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 3406.056196] Key type lgssc registered [ 3406.625359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3409.536766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3409.542613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.693639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.931534] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3414.750726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.865493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3416.127200] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3417.895601] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3419.024420] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3421.623983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.697289] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3421.698427] LustreError: 184068:0:(llog.c:1687:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3421.700738] Lustre: 184068:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.202.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3421.731988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.554358] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3425.318171] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3429.858555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.859771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.861776] Lustre: Skipped 2 previous similar messages [ 3429.864406] Lustre: Skipped 2 previous similar messages [ 3432.151173] Lustre: server umount lustre-MDT0000 complete [ 3433.342052] LustreError: 182413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426650 with bad export cookie 5175364885067557431 [ 3433.345066] LustreError: 182413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3433.345735] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.451426] Lustre: server umount lustre-MDT0001 complete [ 3444.944879] Lustre: server umount lustre-OST0000 complete [ 3447.823530] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3450.908907] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3462.926487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.026112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3463.035850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3463.079061] Lustre: lustre-MDT0000: new disk, initializing [ 3463.102222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.111381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3464.740497] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3469.109767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.173769] Lustre: 187440:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3469.206529] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3469.208998] Lustre: Skipped 1 previous similar message [ 3469.211625] LustreError: 188207:0:(llog.c:1687:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3469.216974] LustreError: 188207:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3469.220994] Lustre: 188207:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3469.233813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3469.237593] Lustre: Skipped 1 previous similar message [ 3469.280341] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3469.304632] Lustre: lustre-MDT0001: new disk, initializing [ 3469.362580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3469.384757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3469.392350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3470.789819] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3473.792927] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3475.041598] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3477.292737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.402982] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3477.404497] Lustre: Skipped 1 previous similar message [ 3477.405589] LustreError: 189161:0:(llog.c:1687:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3477.409669] LustreError: 189161:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3477.412911] Lustre: 189161:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3477.418242] Lustre: 189161:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3477.429710] Lustre: lustre-OST0000: new disk, initializing [ 3477.431443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3477.458927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.907826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3478.912039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3478.944996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3479.289711] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3482.300634] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3483.613859] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3483.683666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3484.994903] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3485.070540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3487.763429] Lustre: server umount lustre-OST0000 complete [ 3489.250574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.251747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.256189] Lustre: Skipped 1 previous similar message [ 3489.262540] Lustre: Skipped 1 previous similar message [ 3494.369115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3494.370877] Lustre: Skipped 1 previous similar message [ 3499.489276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3499.491423] Lustre: Skipped 1 previous similar message [ 3499.519521] Lustre: server umount lustre-MDT0000 complete [ 3500.730131] LustreError: 187431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426718 with bad export cookie 5175364885067558117 [ 3500.732702] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.736288] LustreError: 187431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3511.317692] Lustre: server umount lustre-MDT0001 complete [ 3514.129273] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:38:50 (1777426730) [ 3519.343735] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3522.716293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.853098] LustreError: 192144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.859254] LustreError: 192144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3522.869379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3524.137830] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3526.883792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.274279] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3529.303669] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3531.620405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.747754] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3536.667618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.732765] Lustre: lustre-OST0001: new disk, initializing [ 3536.735385] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3538.734985] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3542.526224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3542.529304] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3542.557733] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3544.239909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3551.061357] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3556.735231] Lustre: DEBUG MARKER: ADD OST3 [ 3558.460703] loop: module loaded [ 3558.483616] loop0: detected capacity change from 0 to 204800000 [ 3558.559837] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3558.562915] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3558.569731] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3560.680574] loop0: detected capacity change from 0 to 204800000 [ 3562.613860] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3562.690250] Lustre: lustre-OST0003: new disk, initializing [ 3562.691994] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3562.732913] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3562.735098] Lustre: Skipped 3 previous similar messages [ 3563.494551] Lustre: DEBUG MARKER: STOP OST3 [ 3563.727984] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3563.769825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3563.774429] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3563.785333] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3564.132435] Lustre: Failing over lustre-OST0003 [ 3564.180634] Lustre: server umount lustre-OST0003 complete [ 3564.402429] LustreError: 194902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3564.668111] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3565.792129] LustreError: 196359:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3567.464546] LustreError: 193594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3568.702827] Lustre: DEBUG MARKER: START OST3 again [ 3569.120915] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3569.125074] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.511873] LustreError: 194902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.519104] LustreError: 194902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3570.920586] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.998778] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3572.652712] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3572.662110] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3572.662161] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3572.833079] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3586.168501] Lustre: Failing over lustre-OST0003 [ 3586.207099] Lustre: server umount lustre-OST0003 complete [ 3586.535707] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.538782] LustreError: 198166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.539158] Lustre: Skipped 2 previous similar messages [ 3586.548884] LustreError: 198166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3596.774896] LustreError: 194902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.780228] LustreError: 194902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3598.817481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3598.821567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.826102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.832175] Lustre: server umount lustre-MDT0000 complete [ 3606.083858] LustreError: 192126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426823 with bad export cookie 5175364885067558971 [ 3606.086713] LustreError: 192126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3606.086978] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3607.010995] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.014697] Lustre: Skipped 4 previous similar messages [ 3612.415545] Lustre: server umount lustre-MDT0001 complete [ 3613.793346] Lustre: server umount lustre-OST0000 complete [ 3615.247721] Lustre: server umount lustre-OST0001 complete [ 3617.412078] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3620.091168] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3633.487821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.580833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.591907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.632511] Lustre: lustre-MDT0000: new disk, initializing [ 3633.656462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.658885] Lustre: Skipped 1 previous similar message [ 3633.663880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3634.988902] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3639.269077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.313668] Lustre: 201465:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3639.425731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3640.724337] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3643.816650] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3644.925609] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3647.174964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3648.636352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3648.671357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3649.231934] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3652.499343] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3653.875397] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3653.942442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3655.226392] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3655.302395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3659.233736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3659.235846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.236944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3659.239040] Lustre: Skipped 1 previous similar message [ 3659.240751] Lustre: Skipped 8 previous similar messages [ 3662.032416] Lustre: server umount lustre-OST0000 complete [ 3669.402718] Lustre: server umount lustre-MDT0000 complete [ 3669.473038] LustreError: 201477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3669.476972] LustreError: 201477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3670.610682] LustreError: 201457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426887 with bad export cookie 5175364885067561246 [ 3670.615138] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.617527] LustreError: 201457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.728202] Lustre: server umount lustre-MDT0001 complete [ 3671.906690] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:41:28 (1777426888) [ 3677.148222] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3680.168112] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3692.602364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.687983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3692.697900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3692.699513] Lustre: Skipped 3 previous similar messages [ 3692.736183] Lustre: lustre-MDT0000: new disk, initializing [ 3692.737795] Lustre: Skipped 2 previous similar messages [ 3692.764438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3692.766580] Lustre: Skipped 2 previous similar messages [ 3694.049109] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3698.255281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.303842] Lustre: 207629:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3698.403765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3699.892541] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3703.110660] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3704.295053] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3706.529035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3707.749450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3708.607609] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3711.772963] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3713.064372] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3713.129809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3714.410860] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3714.485424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3718.114635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3718.116220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.118168] LustreError: Skipped 2 previous similar messages [ 3718.118956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.124084] Lustre: Skipped 3 previous similar messages [ 3718.130342] Lustre: Skipped 3 previous similar messages [ 3721.169141] Lustre: server umount lustre-OST0000 complete [ 3728.532651] Lustre: server umount lustre-MDT0000 complete [ 3729.726144] LustreError: 207623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426947 with bad export cookie 5175364885067562128 [ 3729.730043] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.730215] LustreError: 207623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3729.846449] Lustre: server umount lustre-MDT0001 complete [ 3732.599192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.751695] LustreError: 211435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3734.044705] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3736.898692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.285979] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3740.057537] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3741.198966] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3743.428212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3745.394512] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3747.131461] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3768.292081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.294582] LustreError: Skipped 1 previous similar message [ 3770.270369] Lustre: server umount lustre-OST0000 complete [ 3777.782000] Lustre: server umount lustre-MDT0000 complete [ 3779.308757] LustreError: 211416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426996 with bad export cookie 5175364885067563010 [ 3779.313520] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3779.314725] LustreError: 211416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3779.439107] Lustre: server umount lustre-MDT0001 complete [ 3879.919429] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 3881.006784] Key type lgssc unregistered [ 3881.138309] LNet: 214766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3881.140507] LNetError: 214766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3881.149275] LNet: Removed LNI 192.168.202.149@tcp [ 3881.446106] Key type .llcrypt unregistered [ 3881.447156] Key type ._llcrypt unregistered [ 3889.570487] Key type ._llcrypt registered [ 3889.572931] Key type .llcrypt registered [ 3889.619673] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3895.614929] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3896.110565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3896.120028] alg: No test for adler32 (adler32-zlib) [ 3896.974208] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 3897.052526] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 3898.632121] Key type lgssc registered [ 3898.994422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3911.293317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3911.298503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.390347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3912.404725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3912.452858] Lustre: lustre-MDT0000: new disk, initializing [ 3912.479261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.490053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3913.940238] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3918.025074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.062253] Lustre: 218057:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3918.083807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3918.085672] Lustre: Skipped 1 previous similar message [ 3918.120796] Lustre: lustre-MDT0001: new disk, initializing [ 3918.153807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3918.166966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3918.169835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3919.483955] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3922.530424] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.712793] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3926.015176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3926.125864] Lustre: lustre-OST0000: new disk, initializing [ 3926.128141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3926.157788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.352829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3927.357187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3927.396169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3927.991038] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3931.229786] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3932.691475] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3932.764325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3934.044883] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3934.114504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3937.761496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.762180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.763745] LustreError: Skipped 1 previous similar message [ 3937.764610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.769283] Lustre: Skipped 1 previous similar message [ 3942.882861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3942.885768] Lustre: Skipped 2 previous similar messages [ 3948.002581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3948.004954] Lustre: Skipped 1 previous similar message [ 3949.024093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3949.075208] Lustre: server umount lustre-OST0000 complete [ 3951.009787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.010273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3951.012931] Lustre: Skipped 1 previous similar message [ 3956.193094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.194871] Lustre: Skipped 2 previous similar messages [ 3956.440099] Lustre: server umount lustre-MDT0000 complete [ 3957.609256] LustreError: 219798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427174 with bad export cookie 18206351899330478817 [ 3957.611249] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.614172] LustreError: 219798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3957.725361] Lustre: server umount lustre-MDT0001 complete [ 3963.194289] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3966.411029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3966.564325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.887240] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3970.766377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.886678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3972.140894] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3973.108306] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3975.388074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.353230] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3980.046454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3980.103947] Lustre: lustre-OST0001: new disk, initializing [ 3980.106387] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3980.134979] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.137357] Lustre: Skipped 1 previous similar message [ 3980.829754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3980.834527] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3980.883351] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3982.343253] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3987.660183] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3991.237994] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3999.945741] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:46:56 (1777427216) [ 4001.761065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4001.764516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.768284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.872645] Lustre: server umount lustre-MDT0000 complete [ 4007.907399] LustreError: 222557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.912358] LustreError: 222557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4008.070098] LustreError: 225651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427225 with bad export cookie 18206351899330479685 [ 4008.073072] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.073651] LustreError: 225651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4008.186883] Lustre: server umount lustre-MDT0001 complete [ 4019.199675] Lustre: server umount lustre-OST0000 complete [ 4030.651961] Lustre: server umount lustre-OST0001 complete [ 4032.701970] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4036.314611] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4048.443106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.532724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4048.545283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4048.581912] Lustre: lustre-MDT0000: new disk, initializing [ 4048.606975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4048.612904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4049.892562] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4054.233569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.280296] Lustre: 228707:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4054.382820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4055.663327] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4058.728406] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4059.847596] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4062.004687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4062.108446] Lustre: lustre-OST0000: new disk, initializing [ 4062.109731] Lustre: Skipped 1 previous similar message [ 4062.111386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4062.113465] Lustre: Skipped 2 previous similar messages [ 4063.802170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4063.804434] Lustre: Skipped 1 previous similar message [ 4063.806193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4063.840374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4063.922395] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4066.970447] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4068.490389] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4068.582987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4069.919931] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4069.985823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4073.954882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4073.955383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.957596] LustreError: Skipped 1 previous similar message [ 4073.961230] Lustre: Skipped 3 previous similar messages [ 4073.961857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4073.965979] Lustre: Skipped 5 previous similar messages [ 4076.691336] Lustre: server umount lustre-OST0000 complete [ 4079.073742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.077014] Lustre: Skipped 2 previous similar messages [ 4092.384105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4092.454048] Lustre: server umount lustre-MDT0000 complete [ 4093.672065] LustreError: 228699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427310 with bad export cookie 18206351899330481785 [ 4093.675055] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.675798] LustreError: 228699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4093.790467] Lustre: server umount lustre-MDT0001 complete [ 4096.517379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4096.679494] LustreError: 232518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.707257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.711642] Lustre: Skipped 2 previous similar messages [ 4098.061106] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4100.882500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.326812] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4104.235647] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4105.381532] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.520721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.391330] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4111.362580] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4254.177207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4254.181228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.188516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4254.191796] Lustre: Skipped 6 previous similar messages [ 4260.197401] Lustre: server umount lustre-OST0000 complete [ 4267.606069] Lustre: server umount lustre-MDT0000 complete [ 4268.514424] LustreError: 234646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.519224] LustreError: 234646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4268.850638] LustreError: 232499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427486 with bad export cookie 18206351899330482667 [ 4268.854240] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.856458] LustreError: 232499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4268.982355] Lustre: server umount lustre-MDT0001 complete [ 4375.201237] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 4376.187841] Key type lgssc unregistered [ 4376.313385] LNet: 235773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4376.316595] LNetError: 235773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4376.330492] LNet: Removed LNI 192.168.202.149@tcp [ 4376.728134] Key type .llcrypt unregistered [ 4376.730193] Key type ._llcrypt unregistered [ 4384.949567] Key type ._llcrypt registered [ 4384.950859] Key type .llcrypt registered [ 4384.994992] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4391.044319] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4391.369117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4391.419452] alg: No test for adler32 (adler32-zlib) [ 4392.264106] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4392.341782] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 4393.920249] Key type lgssc registered [ 4394.277637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4408.498528] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4411.907805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4411.915223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.007151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4413.018883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4413.054611] Lustre: lustre-MDT0000: new disk, initializing [ 4413.077907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.086274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4414.347905] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4418.329977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.367379] Lustre: 239762:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4418.392809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4418.395672] Lustre: Skipped 1 previous similar message [ 4418.431711] Lustre: lustre-MDT0001: new disk, initializing [ 4418.466732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4418.479132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4418.482757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4419.694728] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4422.050358] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4424.173840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4424.291041] Lustre: lustre-OST0000: new disk, initializing [ 4424.294339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4424.331867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.116948] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4427.575788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4427.580655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4427.618052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4430.122012] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4430.189146] Lustre: lustre-OST0001: new disk, initializing [ 4430.193075] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4430.228310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.084502] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4435.464196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4435.467160] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4435.495515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4437.478348] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4440.131458] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4449.249196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4449.252411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.256870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.786873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.786995] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.790182] Lustre: Skipped 1 previous similar message [ 4450.793836] Lustre: Skipped 2 previous similar messages [ 4454.058921] Lustre: server umount lustre-MDT0000 complete [ 4455.257826] LustreError: 239754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427672 with bad export cookie 8762803838778944645 [ 4455.262853] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.387588] Lustre: server umount lustre-MDT0001 complete [ 4466.954546] Lustre: server umount lustre-OST0000 complete [ 4478.638468] Lustre: server umount lustre-OST0001 complete [ 4483.686342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4487.253825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4490.165293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4493.303670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4499.364334] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4502.629158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.636244] Lustre: lustre-MDT0000: reset Object Index mappings [ 4502.811493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.057096] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4506.852098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.860504] Lustre: lustre-MDT0001: reset Object Index mappings [ 4508.241265] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4509.446300] Lustre: 248289:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4511.955940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4512.094732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.096483] Lustre: Skipped 1 previous similar message [ 4513.128125] LustreError: 248643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.136601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4513.142025] LustreError: 248643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4514.223164] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4517.222374] LustreError: 248802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.232985] LustreError: 248802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4517.364484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.221935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4519.315290] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4522.492078] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4523.651078] Lustre: 250130:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4532.713542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.716174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4532.716904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4532.721886] Lustre: Skipped 1 previous similar message [ 4536.118908] Lustre: server umount lustre-MDT0000 complete [ 4537.343110] LustreError: 247162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427754 with bad export cookie 8762803838778951421 [ 4537.346043] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4537.346520] LustreError: 247162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4537.458679] Lustre: server umount lustre-MDT0001 complete [ 4548.932053] Lustre: server umount lustre-OST0000 complete [ 4560.484268] Lustre: server umount lustre-OST0001 complete [ 4566.945890] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:56:23 (1777427783) [ 4572.248041] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4576.153437] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4588.972036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.080936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4589.093849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4589.131644] Lustre: lustre-MDT0000: new disk, initializing [ 4589.156337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.158254] Lustre: Skipped 1 previous similar message [ 4589.163833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4590.608602] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4595.503969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.540527] Lustre: 254346:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4595.635419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4597.013933] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4600.169134] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4601.321270] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4603.594902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4603.703082] Lustre: lustre-OST0000: new disk, initializing [ 4603.705375] Lustre: Skipped 1 previous similar message [ 4603.708255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4603.711361] Lustre: Skipped 2 previous similar messages [ 4605.567049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4605.569511] Lustre: Skipped 1 previous similar message [ 4605.571116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4605.612416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4605.635762] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4608.673149] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.108870] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4610.220443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4611.592787] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4611.658821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4614.368602] Lustre: server umount lustre-OST0000 complete [ 4616.161653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4616.162319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.166720] Lustre: Skipped 1 previous similar message [ 4616.168613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4616.170778] Lustre: Skipped 4 previous similar messages [ 4621.281127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4621.283410] Lustre: Skipped 1 previous similar message [ 4625.404374] Lustre: server umount lustre-MDT0000 complete [ 4626.402058] LustreError: 255132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.406027] LustreError: 255132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4626.610150] LustreError: 255124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427843 with bad export cookie 8762803838778953514 [ 4626.613263] LustreError: 255124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4626.613470] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4636.178219] Lustre: server umount lustre-MDT0001 complete [ 4641.801776] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4645.148811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.288327] LustreError: 258851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.302656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.305138] Lustre: Skipped 2 previous similar messages [ 4646.575176] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4649.390994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.760467] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4651.684034] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4653.767762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.681107] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4658.504662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.569834] Lustre: lustre-OST0001: new disk, initializing [ 4658.571800] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4659.932172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4659.939298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4659.995063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4660.791813] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4665.213867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4666.418237] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4673.199300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4673.206532] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4675.041269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4675.044890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.050760] Lustre: Skipped 1 previous similar message [ 4675.053397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4681.202170] Lustre: server umount lustre-MDT0000 complete [ 4682.404382] LustreError: 258832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427899 with bad export cookie 8762803838778954396 [ 4682.407889] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4682.409285] LustreError: 258832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4682.530396] Lustre: server umount lustre-MDT0001 complete [ 4694.021611] Lustre: server umount lustre-OST0000 complete [ 4705.506364] Lustre: server umount lustre-OST0001 complete [ 4710.553197] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4713.790217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.934751] LustreError: 264129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.954154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4713.955906] Lustre: Skipped 3 previous similar messages [ 4715.220916] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4718.008199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.402163] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4720.340417] Lustre: 265236:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4722.502729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4723.685437] LustreError: 265589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.689838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4724.485602] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4727.195311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4728.301810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4729.091938] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4732.093557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4733.239841] Lustre: 267084:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4739.910514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4739.923363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4742.775761] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:59:19 (1777427959) [ 4759.012776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.014184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4759.015834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4759.015846] Lustre: Skipped 7 previous similar messages [ 4759.015943] Lustre: Skipped 4 previous similar messages [ 4760.412285] Lustre: server umount lustre-MDT0000 complete [ 4761.574983] LustreError: 264110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427978 with bad export cookie 8762803838778960689 [ 4761.578245] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4761.580055] LustreError: 264110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4764.129679] LustreError: 265610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4764.134029] LustreError: 265610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4767.743480] Lustre: server umount lustre-MDT0001 complete [ 4775.105269] Lustre: server umount lustre-OST0000 complete [ 4782.357609] Lustre: server umount lustre-OST0001 complete [ 4784.250081] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4787.228087] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4801.424721] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4804.889839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.978282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4804.990316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4805.034097] Lustre: lustre-MDT0000: new disk, initializing [ 4805.064876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4806.320591] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4810.321838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.358613] Lustre: 271262:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4810.448730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4811.669155] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4815.991047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4816.365921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4817.935880] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4822.037946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4823.525935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4823.965099] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4828.427505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4838.662097] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 22:00:55 (1777428055) [ 4841.440826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4841.442757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.445687] Lustre: Skipped 3 previous similar messages [ 4841.448307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4841.449786] Lustre: Skipped 7 previous similar messages [ 4845.541481] Lustre: server umount lustre-MDT0000 complete [ 4846.802593] LustreError: 271256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428064 with bad export cookie 8762803838778968557 [ 4846.805987] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4846.808261] LustreError: 271256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4846.934531] Lustre: server umount lustre-MDT0001 complete [ 4858.435258] Lustre: server umount lustre-OST0000 complete [ 4868.893459] Lustre: server umount lustre-OST0001 complete [ 4874.654148] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4877.574735] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4888.911294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.991596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4888.996062] Lustre: Skipped 2 previous similar messages [ 4889.004210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4889.005647] Lustre: Skipped 4 previous similar messages [ 4889.038522] Lustre: lustre-MDT0000: new disk, initializing [ 4889.039691] Lustre: Skipped 3 previous similar messages [ 4889.060609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.062304] Lustre: Skipped 7 previous similar messages [ 4889.066965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4889.068942] Lustre: Skipped 3 previous similar messages [ 4890.214862] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4894.143314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4894.182362] Lustre: 278304:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4894.276099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4894.278465] Lustre: Skipped 2 previous similar messages [ 4895.486202] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4898.397895] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4899.534146] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4901.791557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4903.191171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4903.807372] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4906.782157] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4907.441841] Lustre: Mounted lustre-client [ 4907.738944] LustreError: 280801:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff936343128800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4907.763153] Lustre: Unmounted lustre-client [ 4912.612385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.612951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4912.616421] Lustre: Skipped 4 previous similar messages [ 4912.619937] Lustre: Skipped 5 previous similar messages [ 4916.686051] Lustre: server umount lustre-OST0000 complete [ 4919.776827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4924.067225] Lustre: server umount lustre-MDT0000 complete [ 4925.298594] LustreError: 279091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428142 with bad export cookie 8762803838778970083 [ 4925.301733] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.304095] LustreError: 279091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4925.446593] Lustre: server umount lustre-MDT0001 complete [ 4928.886678] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 4929.945759] Key type lgssc unregistered [ 4959.648101] LNet: 1 local NIs in recovery (showing 1): 192.168.202.149@tcp [ 4959.776363] LNet: 282079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4959.779262] LNetError: 282079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5039.654334] LNet: Removed LNI 192.168.202.149@tcp [ 5039.942097] Key type .llcrypt unregistered [ 5039.943056] Key type ._llcrypt unregistered [ 5045.540760] Key type ._llcrypt registered [ 5045.541654] Key type .llcrypt registered [ 5045.572406] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 5051.788107] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5052.168500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5052.177503] alg: No test for adler32 (adler32-zlib) [ 5053.032466] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5053.123978] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 5054.704164] Key type lgssc registered [ 5055.056646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5064.332373] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:04:40 (1777428280) [ 5066.564345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5066.569800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.655591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5067.669968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5067.707455] Lustre: lustre-MDT0000: new disk, initializing [ 5067.729915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5067.735466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5069.029891] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5073.516852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.552830] Lustre: 285236:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5073.573967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5073.575499] Lustre: Skipped 1 previous similar message [ 5073.610125] Lustre: lustre-MDT0001: new disk, initializing [ 5073.638621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5073.649053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5073.651996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5074.852931] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5077.870497] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5079.053569] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5081.305469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5081.400882] Lustre: lustre-OST0000: new disk, initializing [ 5081.402796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5081.430753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.276498] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5084.663709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5084.666744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5084.698346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5086.423235] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5094.885161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.885316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.895099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5094.899178] LustreError: Skipped 1 previous similar message [ 5096.401464] Lustre: server umount lustre-OST0000 complete [ 5099.488897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5099.490909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.494173] Lustre: Skipped 1 previous similar message [ 5099.495609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5099.497823] Lustre: Skipped 1 previous similar message [ 5103.775072] Lustre: server umount lustre-MDT0000 complete [ 5105.004978] LustreError: 286017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428322 with bad export cookie 5753356077506912785 [ 5105.007034] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.008007] LustreError: 286017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5105.121810] Lustre: server umount lustre-MDT0001 complete [ 5108.625295] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5109.583050] Key type lgssc unregistered [ 5109.698324] LNet: 288610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5109.700589] LNetError: 288610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5109.711273] LNet: Removed LNI 192.168.202.149@tcp [ 5109.981287] Key type .llcrypt unregistered [ 5109.982231] Key type ._llcrypt unregistered [ 5119.324019] Key type ._llcrypt registered [ 5119.325419] Key type .llcrypt registered [ 5119.364256] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5119.679800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5119.732087] alg: No test for adler32 (adler32-zlib) [ 5120.580405] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5120.663839] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 5122.248110] Key type lgssc registered [ 5122.593265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5125.101916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5125.109053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.265725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.499410] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5130.272215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.385585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5131.678217] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5133.316626] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5134.449651] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5136.514551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5136.618483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.337767] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5140.028657] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5143.541576] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5144.036851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5145.368833] Lustre: server umount lustre-MDT0000 complete [ 5146.080721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5146.082768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.086121] LustreError: 289988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5146.089859] LustreError: 289988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5146.509088] LustreError: 289968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428363 with bad export cookie 5375739570935283377 [ 5146.512747] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.664949] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.49@tcp (stopping) [ 5146.669802] LustreError: 291769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5150.177192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.177599] LustreError: 291769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5150.180406] Lustre: Skipped 1 previous similar message [ 5150.180816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5151.201173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5152.678170] Lustre: server umount lustre-MDT0001 complete [ 5156.913698] Lustre: lustre-OST0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 5156.917903] Lustre: Skipped 1 previous similar message [ 5162.022695] Lustre: lustre-OST0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 5168.096063] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5168.152590] Lustre: server umount lustre-OST0000 complete [ 5170.886620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.249592] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5173.186438] Lustre: 293715:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.149@tcp [ 5173.893984] Lustre: server umount MGS complete [ 5176.757546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.908699] LustreError: 294285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.913648] LustreError: 294285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5176.927942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.179811] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5180.872628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.996377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5182.190393] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5184.885765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5186.835270] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5188.564843] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5199.443787] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 5200.489212] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5203.044534] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5206.497925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5206.499488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5206.502534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5206.507105] Lustre: Skipped 1 previous similar message [ 5206.511411] Lustre: Skipped 2 previous similar messages [ 5210.250868] Lustre: server umount lustre-OST0000 complete [ 5211.617664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.617755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5211.623993] Lustre: Skipped 1 previous similar message [ 5217.618596] Lustre: server umount lustre-MDT0000 complete [ 5218.814030] LustreError: 296753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428436 with bad export cookie 5375739570935284203 [ 5218.816753] LustreError: 296753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5218.816846] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5218.926622] Lustre: server umount lustre-MDT0001 complete [ 5222.430954] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5223.322307] Key type lgssc unregistered [ 5223.437350] LNet: 297486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5223.439449] LNetError: 297486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5223.447256] LNet: Removed LNI 192.168.202.149@tcp [ 5223.700915] Key type .llcrypt unregistered [ 5223.702423] Key type ._llcrypt unregistered [ 5229.034728] Key type ._llcrypt registered [ 5229.035658] Key type .llcrypt registered [ 5229.067095] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 5234.996036] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5235.416751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5235.426575] alg: No test for adler32 (adler32-zlib) [ 5236.272744] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5236.351166] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 5237.928143] Key type lgssc registered [ 5238.256157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5249.255284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5249.262043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.365552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5250.382764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5250.439255] Lustre: lustre-MDT0000: new disk, initializing [ 5250.471205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.481938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5251.761599] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5256.040951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.085586] Lustre: 300582:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5256.106256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5256.107962] Lustre: Skipped 1 previous similar message [ 5256.142125] Lustre: lustre-MDT0001: new disk, initializing [ 5256.169772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5256.179461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5256.182350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5257.459688] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5260.410639] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5261.574616] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5263.694415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5263.799673] Lustre: lustre-OST0000: new disk, initializing [ 5263.801694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5263.827240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.483806] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5267.064597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5267.067347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5267.107511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5268.395621] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5269.606032] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5269.668465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5270.903433] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5270.963928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5272.546485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5272.547471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.548833] LustreError: Skipped 1 previous similar message [ 5272.549533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5272.552809] Lustre: Skipped 1 previous similar message [ 5272.557248] Lustre: Skipped 1 previous similar message [ 5273.614984] Lustre: server umount lustre-OST0000 complete [ 5276.640986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5276.644039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.649323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5277.665420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5277.665819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.920422] Lustre: server umount lustre-MDT0000 complete [ 5282.122341] LustreError: 300573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428499 with bad export cookie 10875816995743015164 [ 5282.125239] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5282.127978] LustreError: 300573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5282.239513] Lustre: server umount lustre-MDT0001 complete [ 5284.938591] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:08:21 (1777428501) [ 5289.928634] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 5292.951605] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5304.452517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.541246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5304.552394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5304.588449] Lustre: lustre-MDT0000: new disk, initializing [ 5304.611434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.617445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5305.916479] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5310.081895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.124673] Lustre: 306888:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5310.144851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5310.146569] Lustre: Skipped 1 previous similar message [ 5310.181650] Lustre: lustre-MDT0001: new disk, initializing [ 5310.214283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5310.225861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5310.229667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5311.480931] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5314.415527] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5315.579730] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5317.810348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5319.778930] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5320.887235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5320.889817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5320.917782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5322.786613] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5336.546650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5336.547118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.547651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5336.547655] Lustre: Skipped 1 previous similar message [ 5336.550052] LustreError: Skipped 1 previous similar message [ 5336.553031] Lustre: Skipped 1 previous similar message [ 5338.266879] Lustre: server umount lustre-OST0000 complete [ 5341.153132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5341.156324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.160841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5341.162458] Lustre: Skipped 1 previous similar message [ 5345.628546] Lustre: server umount lustre-MDT0000 complete [ 5346.767175] LustreError: 306881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428564 with bad export cookie 10875816995743016032 [ 5346.770148] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5346.770238] LustreError: 306881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5346.874387] Lustre: server umount lustre-MDT0001 complete [ 5350.027725] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5351.024197] Key type lgssc unregistered [ 5351.160301] LNet: 310647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5351.162378] LNetError: 310647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5351.168325] LNet: Removed LNI 192.168.202.149@tcp [ 5351.454915] Key type .llcrypt unregistered [ 5351.456478] Key type ._llcrypt unregistered [ 5381.675346] Key type ._llcrypt registered [ 5381.676423] Key type .llcrypt registered [ 5381.713338] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5381.804327] Key type .llcrypt unregistered [ 5381.805326] Key type ._llcrypt unregistered [ 5407.347499] Key type ._llcrypt registered [ 5407.348494] Key type .llcrypt registered [ 5407.382633] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5407.475169] Key type .llcrypt unregistered [ 5407.476755] Key type ._llcrypt unregistered [ 5417.855073] Key type ._llcrypt registered [ 5417.856876] Key type .llcrypt registered [ 5417.895056] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5417.998304] Key type .llcrypt unregistered [ 5417.999246] Key type ._llcrypt unregistered [ 5428.464649] Key type ._llcrypt registered [ 5428.465623] Key type .llcrypt registered [ 5428.502255] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5428.602384] Key type .llcrypt unregistered [ 5428.603405] Key type ._llcrypt unregistered [ 5455.027744] Key type ._llcrypt registered [ 5455.028688] Key type .llcrypt registered [ 5455.061585] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5455.163193] Key type .llcrypt unregistered [ 5455.164118] Key type ._llcrypt unregistered [ 5465.568681] Key type ._llcrypt registered [ 5465.569630] Key type .llcrypt registered [ 5465.600750] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5465.690207] Key type .llcrypt unregistered [ 5465.691213] Key type ._llcrypt unregistered [ 5476.769268] Key type ._llcrypt registered [ 5476.770254] Key type .llcrypt registered [ 5476.813666] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5476.922773] Key type .llcrypt unregistered [ 5476.923839] Key type ._llcrypt unregistered [ 5488.411757] Key type ._llcrypt registered [ 5488.413402] Key type .llcrypt registered [ 5488.461986] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5488.570277] Key type .llcrypt unregistered [ 5488.571204] Key type ._llcrypt unregistered [ 5499.413107] Key type ._llcrypt registered [ 5499.414322] Key type .llcrypt registered [ 5499.451593] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5499.558165] Key type .llcrypt unregistered [ 5499.559185] Key type ._llcrypt unregistered [ 5510.273731] Key type ._llcrypt registered [ 5510.274705] Key type .llcrypt registered [ 5510.313970] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5510.412182] Key type .llcrypt unregistered [ 5510.413237] Key type ._llcrypt unregistered [ 5531.041212] Key type ._llcrypt registered [ 5531.042823] Key type .llcrypt registered [ 5531.080185] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5531.177239] Key type .llcrypt unregistered [ 5531.179045] Key type ._llcrypt unregistered [ 5541.790511] Key type ._llcrypt registered [ 5541.791425] Key type .llcrypt registered [ 5541.824648] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5541.913224] Key type .llcrypt unregistered [ 5541.914136] Key type ._llcrypt unregistered [ 5552.759692] Key type ._llcrypt registered [ 5552.760659] Key type .llcrypt registered [ 5552.796107] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5552.894484] Key type .llcrypt unregistered [ 5552.895898] Key type ._llcrypt unregistered [ 5570.764958] Key type ._llcrypt registered [ 5570.766103] Key type .llcrypt registered [ 5570.805677] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 5577.286698] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5577.633923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5577.641669] alg: No test for adler32 (adler32-zlib) [ 5578.502526] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5578.589347] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 5580.168165] Key type lgssc registered [ 5580.558644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5592.158920] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5592.166627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.270828] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5593.281653] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5593.321724] Lustre: tst250fs-MDT0000: new disk, initializing [ 5593.346045] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.351674] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5594.593692] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5598.572378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.609577] Lustre: 330263:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5598.629581] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5598.631284] Lustre: Skipped 1 previous similar message [ 5598.668224] Lustre: tst250fs-MDT0001: new disk, initializing [ 5598.698170] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5598.708775] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5598.713048] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5599.967493] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5602.195363] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5604.448211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5604.570489] Lustre: tst250fs-OST0000: new disk, initializing [ 5604.572475] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5604.611494] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.472923] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5610.457573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.497110] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5610.501582] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5610.543974] Lustre: tst250fs-OST0001: Not available for connect from 0@lo (not set up) [ 5610.546993] Lustre: tst250fs-OST0001: new disk, initializing [ 5610.549774] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5610.556626] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5610.584900] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.404085] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 5614.903474] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5615.611149] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5615.615409] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5615.624473] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 5620.219563] Lustre: 330265:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.149@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 5625.827941] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.829289] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5625.832739] Lustre: Skipped 3 previous similar messages [ 5625.834644] Lustre: Skipped 2 previous similar messages [ 5630.945518] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5630.947529] Lustre: Skipped 3 previous similar messages [ 5636.066917] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5636.070884] Lustre: Skipped 3 previous similar messages [ 5639.136098] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5639.223467] Lustre: server umount tst250fs-MDT0000 complete [ 5640.761575] LustreError: 331039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428858 with bad export cookie 7095470622452673366 [ 5640.765595] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5640.767191] LustreError: 331039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5640.906476] Lustre: server umount tst250fs-MDT0001 complete [ 5652.357822] Lustre: server umount tst250fs-OST0000 complete [ 5662.741837] Lustre: server umount tst250fs-OST0001 complete [ 5668.458751] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 5670.942012] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5680.816922] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:14:57 (1777428897) [ 5681.318899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5686.721935] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 5689.812374] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5699.685088] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5453 sec ========== 22:15:16 (1777428916) [ 5700.291653] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:15:16 (1777428916) === [ 5701.464408] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:15:17 (1777428917) === [ 5714.136644] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5715.136194] Key type lgssc unregistered [ 5715.263347] LNet: 341710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5715.265496] LNetError: 341710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5715.274292] LNet: Removed LNI 192.168.202.149@tcp [ 5715.583127] Key type .llcrypt unregistered [ 5715.584106] Key type ._llcrypt unregistered