[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 389362118 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002161] x2apic enabled [ 0.002922] Switched APIC routing to physical x2apic. [ 0.004004] kvm-guest: setup PV IPIs [ 0.005720] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006009] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.008068] LSM: Security Framework initializing [ 0.009021] Yama: becoming mindful. [ 0.010016] SELinux: Initializing. [ 0.011033] *** VALIDATE selinux *** [ 0.016541] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.020075] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.021073] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.022057] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023050] *** VALIDATE tmpfs *** [ 0.024275] *** VALIDATE proc *** [ 0.025086] *** VALIDATE cgroup *** [ 0.025596] *** VALIDATE cgroup2 *** [ 0.026148] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.027082] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.028002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.029019] Spectre V2 : User space: Vulnerable [ 0.030003] Speculative Store Bypass: Vulnerable [ 0.032340] debug: unmapping init [mem 0xffffffffb1259000-0xffffffffb1260fff] [ 0.034116] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.035358] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.036008] ... version: 2 [ 0.036694] ... bit width: 48 [ 0.037007] ... generic registers: 4 [ 0.037642] ... value mask: 0000ffffffffffff [ 0.038005] ... max period: 00007fffffffffff [ 0.038913] ... fixed-purpose events: 3 [ 0.039005] ... event mask: 000000070000000f [ 0.039953] rcu: Hierarchical SRCU implementation. [ 0.040908] smp: Bringing up secondary CPUs ... [ 0.041345] x86: Booting SMP configuration: [ 0.042009] .... node #0, CPUs: #1 #2 #3 [ 0.044188] smp: Brought up 1 node, 4 CPUs [ 0.045923] smpboot: Max logical packages: 1 [ 0.046006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.118426] node 0 deferred pages initialised in 71ms [ 0.121189] devtmpfs: initialized [ 0.122178] x86/mm: Memory block size: 128MB [ 0.124502] gcov: version magic: 0x41383552 [ 0.125354] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.126049] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.127154] pinctrl core: initialized pinctrl subsystem [ 0.128093] [ 0.128433] ************************************************************* [ 0.129000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131010] ** ** [ 0.132006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.134005] ** ** [ 0.135005] ** This means that this kernel is built to expose internal ** [ 0.136004] ** IOMMU data structures, which may compromise security on ** [ 0.138005] ** your system. ** [ 0.139004] ** ** [ 0.141005] ** If you see this message and you are not debugging the ** [ 0.142004] ** kernel, report this immediately to your vendor! ** [ 0.143005] ** ** [ 0.145005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146005] ************************************************************* [ 0.147458] NET: Registered protocol family 16 [ 0.149330] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.150022] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.152033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.154075] cpuidle: using governor menu [ 0.155371] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.157289] PCI: Using configuration type 1 for base access [ 0.158088] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.165124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.167027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.169092] cryptd: max_cpu_qlen set to 1000 [ 0.171172] ACPI: Added _OSI(Module Device) [ 0.172062] ACPI: Added _OSI(Processor Device) [ 0.173007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.174007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.178764] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182469] ACPI: Interpreter enabled [ 0.184041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.185006] ACPI: Using IOAPIC for interrupt routing [ 0.186052] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.188287] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.197622] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.199018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.201009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.203043] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.207170] acpiphp: Slot [2] registered [ 0.208068] acpiphp: Slot [5] registered [ 0.210063] acpiphp: Slot [6] registered [ 0.211078] acpiphp: Slot [7] registered [ 0.212063] acpiphp: Slot [8] registered [ 0.213067] acpiphp: Slot [9] registered [ 0.214071] acpiphp: Slot [10] registered [ 0.215056] acpiphp: Slot [3] registered [ 0.216057] acpiphp: Slot [4] registered [ 0.217056] acpiphp: Slot [11] registered [ 0.218129] acpiphp: Slot [12] registered [ 0.219080] acpiphp: Slot [13] registered [ 0.219900] acpiphp: Slot [14] registered [ 0.221044] acpiphp: Slot [15] registered [ 0.221951] acpiphp: Slot [16] registered [ 0.223043] acpiphp: Slot [17] registered [ 0.223976] acpiphp: Slot [18] registered [ 0.225043] acpiphp: Slot [19] registered [ 0.225897] acpiphp: Slot [20] registered [ 0.226042] acpiphp: Slot [21] registered [ 0.228064] acpiphp: Slot [22] registered [ 0.228944] acpiphp: Slot [23] registered [ 0.229041] acpiphp: Slot [24] registered [ 0.229909] acpiphp: Slot [25] registered [ 0.231051] acpiphp: Slot [26] registered [ 0.231868] acpiphp: Slot [27] registered [ 0.233085] acpiphp: Slot [28] registered [ 0.234045] acpiphp: Slot [29] registered [ 0.234862] acpiphp: Slot [30] registered [ 0.236080] acpiphp: Slot [31] registered [ 0.237004] PCI host bridge to bus 0000:00 [ 0.237895] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.240007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.242009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.243008] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.245010] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.246130] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.248826] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.250786] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.259723] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.262032] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.264008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.265006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.267007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.268507] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.270578] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.272029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.274472] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.277014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.285014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.289009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.293478] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.299006] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.303005] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.313008] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.318833] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.322015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.327011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.339010] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.345618] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.350004] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.354009] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.366012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.373936] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.378007] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.382008] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.391011] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.397939] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.402016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.409017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.418007] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.423913] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.430009] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.434009] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.444009] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.451216] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.452213] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.454236] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.455188] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.457117] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.460131] iommu: Default domain type: Passthrough [ 0.461261] SCSI subsystem initialized [ 0.462053] ACPI: bus type USB registered [ 0.463045] usbcore: registered new interface driver usbfs [ 0.464036] usbcore: registered new interface driver hub [ 0.465038] usbcore: registered new device driver usb [ 0.466114] pps_core: LinuxPPS API ver. 1 registered [ 0.467005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.469017] PTP clock support registered [ 0.470082] EDAC MC: Ver: 3.0.0 [ 0.471290] PCI: Using ACPI for IRQ routing [ 0.472689] NetLabel: Initializing [ 0.474008] NetLabel: domain hash size = 128 [ 0.475006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.477057] NetLabel: unlabeled traffic allowed by default [ 0.478091] vgaarb: loaded [ 0.479208] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.480005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.486203] clocksource: Switched to clocksource kvm-clock [ 0.553307] VFS: Disk quotas dquot_6.6.0 [ 0.554208] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.555481] *** VALIDATE ramfs *** [ 0.556282] *** VALIDATE hugetlbfs *** [ 0.557237] pnp: PnP ACPI init [ 0.558672] pnp: PnP ACPI: found 6 devices [ 0.570164] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.572205] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.573458] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.574635] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.576088] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.577471] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.578860] NET: Registered protocol family 2 [ 0.580395] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.583463] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.585541] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.588948] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.590827] TCP: Hash tables configured (established 65536 bind 65536) [ 0.592521] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.594220] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.595675] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.597169] NET: Registered protocol family 1 [ 0.598415] RPC: Registered named UNIX socket transport module. [ 0.599509] RPC: Registered udp transport module. [ 0.600281] RPC: Registered tcp transport module. [ 0.601193] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.602542] NET: Registered protocol family 44 [ 0.603458] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.604662] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.605995] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.607353] PCI: CLS 0 bytes, default 64 [ 0.608834] Unpacking initramfs... [ 1.649970] debug: unmapping init [mem 0xffff9704fcc54000-0xffff9704fffbffff] [ 1.652126] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.653317] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.654736] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.041620] Initialise system trusted keyrings [ 2.042611] Key type blacklist registered [ 2.043593] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.049555] zbud: loaded [ 2.051366] *** VALIDATE nfs *** [ 2.052161] *** VALIDATE nfs4 *** [ 2.052918] pstore: using deflate compression [ 2.054361] Platform Keyring initialized [ 2.115797] NET: Registered protocol family 38 [ 2.116889] Key type asymmetric registered [ 2.117647] Asymmetric key parser 'x509' registered [ 2.118676] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.120368] io scheduler mq-deadline registered [ 2.121948] io scheduler kyber registered [ 2.123445] io scheduler bfq registered [ 2.124922] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.127480] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.129808] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.132308] ACPI: Power Button [PWRF] [ 2.136894] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.142579] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.151961] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.157563] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.168621] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.195925] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.222253] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.225165] Non-volatile memory driver v1.3 [ 2.226451] Linux agpgart interface v0.103 [ 2.245463] virtio_blk virtio1: [vda] 134696 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.247085] vda: detected capacity change from 0 to 68964352 [ 2.256253] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.258125] vdb: detected capacity change from 0 to 1073741824 [ 2.267278] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.269064] vdc: detected capacity change from 0 to 2621440000 [ 2.279849] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.281558] vdd: detected capacity change from 0 to 2621440000 [ 2.290900] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.292561] vde: detected capacity change from 0 to 4294967296 [ 2.301761] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.303741] vdf: detected capacity change from 0 to 4294967296 [ 2.307431] libphy: Fixed MDIO Bus: probed [ 2.310762] usbcore: registered new interface driver usbserial_generic [ 2.312357] usbserial: USB Serial support registered for generic [ 2.313960] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.316570] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.317507] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.319282] mousedev: PS/2 mouse device common for all mice [ 2.320636] rtc_cmos 00:05: RTC can wake from S4 [ 2.322281] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.324139] rtc_cmos 00:05: registered as rtc0 [ 2.325724] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.325870] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.328392] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.329444] intel_pstate: CPU model not supported [ 2.332597] hid: raw HID events driver (C) Jiri Kosina [ 2.333978] usbcore: registered new interface driver usbhid [ 2.335225] usbhid: USB HID core driver [ 2.336255] drop_monitor: Initializing network drop monitor service [ 2.337682] Initializing XFRM netlink socket [ 2.338989] NET: Registered protocol family 10 [ 2.340766] Segment Routing with IPv6 [ 2.341533] NET: Registered protocol family 17 [ 2.342543] mpls_gso: MPLS GSO support [ 2.347129] RAS: Correctable Errors collector initialized. [ 2.348383] AVX version of gcm_enc/dec engaged. [ 2.349431] AES CTR mode by8 optimization enabled [ 2.386721] sched_clock: Marking stable (2386705023, 0)->(2957067294, -570362271) [ 2.389508] registered taskstats version 1 [ 2.390963] Loading compiled-in X.509 certificates [ 2.392384] zswap: loaded using pool lzo/zbud [ 2.405428] Key type big_key registered [ 2.413143] Key type encrypted registered [ 2.415100] ima: No TPM chip found, activating TPM-bypass! [ 2.416252] ima: Allocated hash algorithm: sha1 [ 2.417258] ima: No architecture policies found [ 2.418196] evm: Initialising EVM extended attributes: [ 2.419273] evm: security.selinux [ 2.419824] evm: security.ima [ 2.420427] evm: security.capability [ 2.421143] evm: HMAC attrs: 0x1 [ 2.422503] rtc_cmos 00:05: setting system clock to 2026-03-02 18:55:58 UTC (1772477758) [ 2.425986] debug: unmapping init [mem 0xffffffffb2203000-0xffffffffb23fffff] [ 2.427427] debug: unmapping init [mem 0xffffffffb0f82000-0xffffffffb1258fff] [ 2.434045] Write protecting the kernel read-only data: 28672k [ 2.435820] debug: unmapping init [mem 0xffffffffaf603000-0xffffffffaf7fffff] [ 2.437319] debug: unmapping init [mem 0xffffffffaff14000-0xffffffffafffffff] [ 2.458583] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.463511] systemd[1]: Detected virtualization kvm. [ 2.464533] systemd[1]: Detected architecture x86-64. [ 2.465454] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.485794] systemd[1]: No hostname configured. [ 2.487303] systemd[1]: Set hostname to . [ 2.488786] random: systemd: uninitialized urandom read (16 bytes read) [ 2.490801] systemd[1]: Initializing machine ID from random generator. [ 2.522333] random: ln: uninitialized urandom read (6 bytes read) [ 2.578233] random: systemd: uninitialized urandom read (16 bytes read) [ 2.579981] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 2.583900] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 2.586065] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 2.945337] device-mapper: uevent: version 1.0.3 [ 2.946482] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 3.514641] random: fast init done [ 3.517692] virtio_net virtio0 ens2: renamed from eth0 [ 3.546118] scsi host0: ata_piix [ 3.554570] scsi host1: ata_piix [ 3.556488] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.559783] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.698456] dracut-initqueue[590]: RTNETLINK answers: File exists [ 7.925051] random: crng init done [ 7.926143] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 8.319046] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.227669] printk: systemd: 26 output lines suppressed due to ratelimiting [ 9.393375] SELinux: Disabled at runtime. [ 9.443187] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.447836] systemd[1]: Detected virtualization kvm. [ 9.448788] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 9.769202] systemd[1]: initrd-switch-root.service: Succeeded. [ 9.771198] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 9.774192] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 9.777816] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 9.779772] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 9.784559] systemd[1]: Starting Journal Service... Starting Journal Service... [ 9.791327] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ 9.827519] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 10.102634] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 10.350903] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.384618] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.454852] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.467598] EDAC sbridge: Ver: 1.1.2 [ 11.486675] Key type dns_resolver registered [ 11.724418] NFS: Registering the id_resolver key type [ 11.725609] Key type id_resolver registered [ 11.726791] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-server login: [ 32.898503] libcfs: loading out-of-tree module taints kernel. [ 32.916545] Key type ._llcrypt registered [ 32.917605] Key type .llcrypt registered [ 32.955488] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 40.101994] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 40.657959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 40.663526] alg: No test for adler32 (adler32-zlib) [ 41.603517] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 41.834989] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 43.431127] Key type lgssc registered [ 43.910431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 49.478490] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 61.558868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 65.776027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 65.783857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 66.869071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 66.879705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 66.915699] Lustre: lustre-MDT0000: new disk, initializing [ 66.937699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 66.943752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 68.094110] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 72.846130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 72.878383] Lustre: 6476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 72.889722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 72.889937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 72.893380] Lustre: Skipped 1 previous similar message [ 72.928094] Lustre: lustre-MDT0001: new disk, initializing [ 72.944987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 72.952325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 72.954217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 74.078640] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 76.223464] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 79.012697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 79.091517] Lustre: lustre-OST0000: new disk, initializing [ 79.093051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 79.108530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 80.717829] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.945168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 83.948702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 83.961729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 85.702378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 85.744476] Lustre: lustre-OST0001: new disk, initializing [ 85.745960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 85.764064] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 87.478489] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.089568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 90.093095] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 90.110870] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 93.013276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 97.522877] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 104.301303] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing check_logdir /tmp/testlogs/ [ 105.695741] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing yml_node [ 107.078778] Lustre: DEBUG MARKER: Client: 2.17.50.195 [ 107.885178] Lustre: DEBUG MARKER: MDS: 2.17.50.195 [ 108.648166] Lustre: DEBUG MARKER: OSS: 2.17.50.195 [ 109.158366] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 2 13:57:45 EST 2026 [ 114.668466] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 115.197169] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 129.503526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 129.506645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 129.512639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 131.039655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 131.039796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 131.041544] Lustre: Skipped 1 previous similar message [ 131.048433] Lustre: Skipped 1 previous similar message [ 136.159723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 136.162458] Lustre: Skipped 5 previous similar messages [ 141.279126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 141.283471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 141.285466] Lustre: Skipped 3 previous similar messages [ 141.353281] Lustre: server umount lustre-MDT0000 complete [ 142.559658] LustreError: 8380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477898 with bad export cookie 11620990103654868314 [ 142.560991] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 142.562895] LustreError: 8380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 146.399668] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.406935] Lustre: Skipped 2 previous similar messages [ 148.705336] Lustre: server umount lustre-MDT0001 complete [ 164.320399] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 164.355115] Lustre: server umount lustre-OST0000 complete [ 180.191111] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 180.234762] Lustre: server umount lustre-OST0001 complete [ 182.123016] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 184.974621] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 196.144819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.237549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.250836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.283891] Lustre: lustre-MDT0000: new disk, initializing [ 196.307355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.313943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.564357] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 201.621180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.661658] Lustre: 14943:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.755680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 202.888250] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 205.796805] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 206.903633] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 209.014213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.114040] Lustre: lustre-OST0000: new disk, initializing [ 209.115409] Lustre: Skipped 1 previous similar message [ 209.117208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.118933] Lustre: Skipped 2 previous similar messages [ 209.142936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.144802] Lustre: Skipped 1 previous similar message [ 210.422327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 210.425916] Lustre: Skipped 1 previous similar message [ 210.427721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.462981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 210.840587] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 213.676818] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 214.912618] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 214.987391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 216.344191] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 216.426344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 219.168592] Lustre: server umount lustre-OST0000 complete [ 220.640526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 220.641164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 220.644616] Lustre: Skipped 1 previous similar message [ 220.647209] Lustre: Skipped 4 previous similar messages [ 230.925250] Lustre: server umount lustre-MDT0000 complete [ 232.294319] LustreError: 14934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477988 with bad export cookie 11620990103654869847 [ 232.300103] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 242.698672] Lustre: server umount lustre-MDT0001 complete [ 244.371379] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:00:00 (1772478000) [ 250.181631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.318733] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.326309] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 250.340545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.521037] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 254.217934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.612963] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 257.316565] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 258.494983] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 259.186508] Lustre: Failing over lustre-MDT0000 [ 259.189339] LustreError: 19438:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 259.191439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.193486] LustreError: 19438:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 259.198286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 259.200860] LustreError: 19438:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 259.203191] Lustre: Skipped 5 previous similar messages [ 259.294328] Lustre: server umount lustre-MDT0000 complete [ 264.352209] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.356229] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 269.791986] LustreError: 19425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.797089] LustreError: 19425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 272.207237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.245669] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 273.495877] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 276.474330] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 277.472395] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 277.475031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 277.479811] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 279.456835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.461084] Lustre: Skipped 2 previous similar messages [ 284.567516] Lustre: server umount lustre-MDT0000 complete [ 284.639919] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.644068] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 285.793318] LustreError: 19388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478041 with bad export cookie 11620990103654871254 [ 285.797165] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.797318] LustreError: 19388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 285.913743] Lustre: server umount lustre-MDT0001 complete [ 288.604636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 288.747046] LustreError: 22399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.769983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 288.772472] Lustre: Skipped 2 previous similar messages [ 289.993162] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 292.644895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.008997] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 295.679138] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 296.770316] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 297.427173] Lustre: Failing over lustre-MDT0000 [ 297.429745] LustreError: 22428:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 297.431205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.431853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.431856] Lustre: Skipped 2 previous similar messages [ 297.433654] LustreError: 22428:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 297.437020] Lustre: Skipped 1 previous similar message [ 297.445826] LustreError: 22428:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 297.529938] Lustre: server umount lustre-MDT0000 complete [ 302.560050] LustreError: 22399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.566233] LustreError: 22399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 310.336063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 310.372044] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.744270] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 314.952320] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 315.872806] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 315.876054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 315.878798] Lustre: Skipped 1 previous similar message [ 315.885204] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 323.152967] Lustre: server umount lustre-MDT0000 complete [ 324.270203] LustreError: 23115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478080 with bad export cookie 11620990103654872185 [ 324.272565] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.273774] LustreError: 23115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 324.395344] Lustre: server umount lustre-MDT0001 complete [ 327.110842] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:01:23 (1772478083) [ 332.163461] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 335.123092] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 347.720700] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 351.382585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.473622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 351.486335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 351.525068] Lustre: lustre-MDT0000: new disk, initializing [ 351.557267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 352.887936] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 357.090356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.135497] Lustre: 28383:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 357.230313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 357.232725] Lustre: Skipped 3 previous similar messages [ 357.243840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 358.525617] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 361.566365] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 362.751134] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 364.887082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 366.532922] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 366.554137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 366.561385] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 366.843239] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 369.828260] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 371.865471] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 376.801379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 376.801718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.803477] LustreError: Skipped 1 previous similar message [ 376.806563] Lustre: Skipped 2 previous similar messages [ 376.808558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 376.810920] Lustre: Skipped 6 previous similar messages [ 378.017091] Lustre: server umount lustre-OST0000 complete [ 385.503249] Lustre: server umount lustre-MDT0000 complete [ 386.778141] LustreError: 28376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478142 with bad export cookie 11620990103654872591 [ 386.782363] LustreError: 28376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 386.782500] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.932518] Lustre: server umount lustre-MDT0001 complete [ 390.583064] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 391.653203] Key type lgssc unregistered [ 391.791493] LNet: 31944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.793541] LNetError: 31944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 391.801265] LNet: Removed LNI 192.168.202.110@tcp [ 392.121133] Key type .llcrypt unregistered [ 392.122443] Key type ._llcrypt unregistered [ 400.872128] Key type ._llcrypt registered [ 400.873052] Key type .llcrypt registered [ 400.905376] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 401.202757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 401.252407] alg: No test for adler32 (adler32-zlib) [ 402.107858] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 402.188200] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 403.767142] Key type lgssc registered [ 404.096492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 406.543413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 406.550304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.718841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.888350] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 411.479020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.597837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.861097] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 414.627014] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 415.729961] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 417.895663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.010896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.776843] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 421.094325] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 421.400774] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 441.711405] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:03:17 (1772478197) [ 442.340980] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 442.935572] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 445.090175] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:03:20 (1772478200) [ 447.975455] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:03:23 (1772478203) [ 448.342546] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 451.303558] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 451.306113] Lustre: Skipped 2 previous similar messages [ 451.877128] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:03:27 (1772478207) [ 459.094777] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 459.097721] Lustre: Skipped 10 previous similar messages [ 459.988402] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:03:35 (1772478215) [ 460.934842] Lustre: *** cfs_fail_loc=131b, val=2*** [ 462.154199] Lustre: *** cfs_fail_loc=131b, val=2*** [ 465.163739] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:03:40 (1772478220) [ 467.414298] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 467.417519] Lustre: Skipped 6 previous similar messages [ 470.308315] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:03:46 (1772478226) [ 477.663812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 477.666311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.671241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.688568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.688796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.689048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.690079] Lustre: Skipped 1 previous similar message [ 480.270361] Lustre: server umount lustre-OST0000 complete [ 483.295966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.299244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.303578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.649450] Lustre: server umount lustre-MDT0000 complete [ 488.845172] LustreError: 36556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478244 with bad export cookie 8565884569715478311 [ 488.848113] LustreError: 36556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 488.848649] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.953728] Lustre: server umount lustre-MDT0001 complete [ 492.554164] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 493.523838] Key type lgssc unregistered [ 493.647592] LNet: 40817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.652320] LNetError: 40817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.660289] LNet: Removed LNI 192.168.202.110@tcp [ 493.931711] Key type .llcrypt unregistered [ 493.933046] Key type ._llcrypt unregistered [ 499.728845] Key type ._llcrypt registered [ 499.730251] Key type .llcrypt registered [ 499.764799] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 506.188929] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 506.511851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 506.538248] alg: No test for adler32 (adler32-zlib) [ 507.384578] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 507.467355] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 509.039168] Key type lgssc registered [ 509.390884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 520.866690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 520.871452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.958174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 521.969129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 522.021976] Lustre: lustre-MDT0000: new disk, initializing [ 522.045000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.052204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 523.214993] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 527.153108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.194421] Lustre: 43909:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 527.218151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 527.220216] Lustre: Skipped 1 previous similar message [ 527.253918] Lustre: lustre-MDT0001: new disk, initializing [ 527.285125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 527.295729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 527.298218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 528.505077] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 531.426668] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 532.539479] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 534.658984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.759696] Lustre: lustre-OST0000: new disk, initializing [ 534.762680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 534.790677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.600420] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 537.844367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 537.847845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 537.875488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 539.597542] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 541.045642] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 541.118231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 542.452054] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 542.523254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 543.201218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 543.201641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.203316] LustreError: Skipped 1 previous similar message [ 543.207735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 543.210971] Lustre: Skipped 1 previous similar message [ 548.321701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.323541] Lustre: Skipped 1 previous similar message [ 549.269130] Lustre: server umount lustre-OST0000 complete [ 552.927937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 552.930261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.933577] Lustre: Skipped 1 previous similar message [ 552.935181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.701704] Lustre: server umount lustre-MDT0000 complete [ 558.101318] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478314 with bad export cookie 1223153462150272204 [ 558.103525] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.107324] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 558.276949] Lustre: server umount lustre-MDT0001 complete [ 561.594446] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:05:17 (1772478317) [ 563.835475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.058360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 565.325224] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 568.089848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.221446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 569.571990] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 571.371907] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 572.617316] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 574.956718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 576.961802] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 578.758919] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 596.061680] LustreError: 50169:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 596.349827] Lustre: Modifying parameter general.timeout=1 in log params [ 597.488242] Lustre: Modifying parameter general.timeout=5 in log params [ 597.490433] Lustre: Skipped 3 previous similar messages [ 599.684751] Lustre: Modifying parameter general.timeout=13 in log params [ 599.686228] Lustre: Skipped 7 previous similar messages [ 603.870748] Lustre: Modifying parameter general.timeout=28 in log params [ 603.872496] Lustre: Skipped 14 previous similar messages [ 612.070847] Lustre: Modifying parameter general.timeout=57 in log params [ 612.072294] Lustre: Skipped 28 previous similar messages [ 628.070188] Lustre: Modifying parameter general.timeout=114 in log params [ 628.071673] Lustre: Skipped 56 previous similar messages [ 635.390423] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:06:31 (1772478391) [ 636.311902] Lustre: 57063:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 637.481404] Lustre: 57260:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 637.486081] Lustre: 57260:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 647.497228] Lustre: 58905:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 647.500166] Lustre: 58905:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 3 previous similar messages [ 647.506751] LustreError: 58905:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 648.725133] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:06:44 (1772478404) [ 654.162665] Lustre: server umount lustre-MDT0000 complete [ 655.267333] LustreError: 47937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478411 with bad export cookie 1223153462150273079 [ 655.271069] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.271917] LustreError: 47937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 656.352203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.352698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 656.355190] Lustre: Skipped 2 previous similar messages [ 656.355525] LustreError: 47951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.357815] Lustre: Skipped 2 previous similar messages [ 656.358845] LustreError: 47951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 657.376597] Lustre: server umount lustre-MDT0001 complete [ 658.579604] Lustre: server umount lustre-OST0000 complete [ 666.645467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.738312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 666.750307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 666.754307] Lustre: Skipped 38 previous similar messages [ 666.832819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.835083] Lustre: Skipped 1 previous similar message [ 668.088296] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 670.779697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.810834] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 670.821599] Lustre: 61429:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 672.130919] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 673.815338] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 674.930367] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 677.165313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.236128] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 679.103626] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 680.714347] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 686.240361] Lustre: 63655:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 691.681344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 691.683654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.685556] LustreError: Skipped 1 previous similar message [ 691.686181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 691.692229] Lustre: Skipped 1 previous similar message [ 694.091828] Lustre: server umount lustre-OST0000 complete [ 696.800914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.801429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 696.805553] Lustre: Skipped 1 previous similar message [ 701.464129] Lustre: server umount lustre-MDT0000 complete [ 701.919911] LustreError: 61442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.923841] LustreError: 61442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 702.641335] LustreError: 62154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478458 with bad export cookie 1223153462150275361 [ 702.644914] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.646142] LustreError: 62154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.752534] Lustre: server umount lustre-MDT0001 complete [ 706.164491] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 707.095410] Key type lgssc unregistered [ 707.214373] LNet: 64844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.216813] LNetError: 64844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.228302] LNet: Removed LNI 192.168.202.110@tcp [ 707.510829] Key type .llcrypt unregistered [ 707.512068] Key type ._llcrypt unregistered [ 716.461657] Key type ._llcrypt registered [ 716.462485] Key type .llcrypt registered [ 716.506785] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 716.868825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 716.911389] alg: No test for adler32 (adler32-zlib) [ 717.771187] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 717.852794] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 719.431090] Key type lgssc registered [ 719.806428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 722.316468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 722.322303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.493647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.797266] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 727.691989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.899976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 729.228676] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 731.031818] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 732.157675] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 734.202157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.359040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.053185] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 737.610251] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 756.179226] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 758.751848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 758.754333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.757894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.264550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.265774] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.267284] Lustre: Skipped 1 previous similar message [ 759.273309] Lustre: Skipped 1 previous similar message [ 764.384708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 764.386484] Lustre: Skipped 3 previous similar messages [ 764.689762] Lustre: server umount lustre-MDT0000 complete [ 765.896359] LustreError: 67843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478521 with bad export cookie 1741766422092329438 [ 765.899356] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.900030] LustreError: 67843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 766.002124] Lustre: server umount lustre-MDT0001 complete [ 786.947692] Lustre: server umount lustre-OST0000 complete [ 795.516401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.610253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 795.622742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 795.719177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.019586] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 799.908229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.944060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 799.954933] Lustre: 70773:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 800.115633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 801.388524] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 803.078784] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 804.231920] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 806.482850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 806.551574] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 808.576149] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 810.345713] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 816.281638] Lustre: 73007:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 816.930898] Lustre: Modifying parameter general.jobid_var=disable in log params [ 822.753541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 822.754569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.755673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 822.756094] LustreError: Skipped 1 previous similar message [ 822.760822] Lustre: Skipped 1 previous similar message [ 824.269412] Lustre: server umount lustre-OST0000 complete [ 825.823770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 825.826184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.872412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.874465] Lustre: Skipped 3 previous similar messages [ 831.643411] Lustre: server umount lustre-MDT0000 complete [ 832.860255] LustreError: 71502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478588 with bad export cookie 1741766422092330229 [ 832.862701] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.863816] LustreError: 71502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 832.983971] Lustre: server umount lustre-MDT0001 complete [ 836.516926] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 837.442933] Key type lgssc unregistered [ 837.567324] LNet: 74194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.569617] LNetError: 74194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.576243] LNet: Removed LNI 192.168.202.110@tcp [ 837.864333] Key type .llcrypt unregistered [ 837.865195] Key type ._llcrypt unregistered [ 846.238377] Key type ._llcrypt registered [ 846.239583] Key type .llcrypt registered [ 846.272583] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 846.554272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 846.596082] alg: No test for adler32 (adler32-zlib) [ 847.439830] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 847.518715] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 849.095123] Key type lgssc registered [ 849.428465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 851.725109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 851.730195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.880269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.096497] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 856.747899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.940804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 858.237853] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 859.965251] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 861.017482] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 863.105635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 863.270580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.003826] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 866.661198] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.534785] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 885.065235] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 885.067338] Lustre: Skipped 1 previous similar message [ 886.119339] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 886.121958] Lustre: Skipped 3 previous similar messages [ 888.275650] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 888.277808] Lustre: Skipped 7 previous similar messages [ 892.561294] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 892.564986] Lustre: Skipped 14 previous similar messages [ 900.805533] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 900.807392] Lustre: Skipped 26 previous similar messages [ 916.944795] Lustre: 83316:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 917.257224] Lustre: Disabling parameter general.jobid_var= in log params [ 917.259043] Lustre: Skipped 43 previous similar messages [ 917.853150] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:11:13 (1772478673) [ 918.399842] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 918.946240] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:11:14 (1772478674) [ 922.591857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 922.594705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.599859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.618827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.619183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.622787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.738467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 928.740186] Lustre: Skipped 2 previous similar messages [ 933.858111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.861039] Lustre: Skipped 1 previous similar message [ 934.367119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 934.419381] Lustre: server umount lustre-OST0000 complete [ 938.976804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 938.977220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.977779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 938.985476] Lustre: Skipped 1 previous similar message [ 941.782258] Lustre: server umount lustre-MDT0000 complete [ 942.996698] LustreError: 75546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478699 with bad export cookie 9084745785465337873 [ 943.001663] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.126215] Lustre: server umount lustre-MDT0001 complete [ 946.616689] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 947.640403] Key type lgssc unregistered [ 947.780309] LNet: 84546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.784311] LNetError: 84546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.793300] LNet: Removed LNI 192.168.202.110@tcp [ 948.079693] Key type .llcrypt unregistered [ 948.080996] Key type ._llcrypt unregistered [ 949.474782] Key type ._llcrypt registered [ 949.476177] Key type .llcrypt registered [ 949.514415] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 950.837390] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules [ 951.124803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 951.131804] alg: No test for adler32 (adler32-zlib) [ 951.991587] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 952.074650] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 953.655140] Key type lgssc registered [ 954.072956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 954.480220] LustreError: 85665:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 962.202111] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 965.433835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.796241] LustreError: 85665:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 975.805703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.005211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.311878] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 981.241612] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:12:17 (1772478737) [ 982.766156] LustreError: 86702:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 982.769055] LustreError: 86702:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 982.771487] LustreError: 86702:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 982.873546] Lustre: server umount lustre-MDT0000 complete [ 987.307752] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 988.310511] Key type lgssc unregistered [ 988.440445] LNet: 88158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.445188] LNetError: 88158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.458294] LNet: Removed LNI 192.168.202.110@tcp [ 988.745653] Key type .llcrypt unregistered [ 988.746671] Key type ._llcrypt unregistered [ 995.786645] Key type ._llcrypt registered [ 995.787781] Key type .llcrypt registered [ 995.824375] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 996.149197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.184323] alg: No test for adler32 (adler32-zlib) [ 997.050635] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 997.130863] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 998.711120] Key type lgssc registered [ 999.059122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1001.518811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1001.525865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.707279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.001428] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1006.840570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.099770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.434333] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1010.259858] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.433309] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1013.709878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1013.921138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.871681] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1017.645623] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1049.508766] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:13:25 (1772478805) [ 1050.096451] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1050.689173] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:13:26 (1772478806) [ 1053.155587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.157893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1053.158788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.161781] Lustre: Skipped 2 previous similar messages [ 1057.813910] Lustre: server umount lustre-MDT0000 complete [ 1058.272284] LustreError: 89341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.276868] LustreError: 89341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1059.071050] LustreError: 89321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478815 with bad export cookie 11306344496284008242 [ 1059.074079] LustreError: 89321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1059.074170] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.193848] Lustre: server umount lustre-MDT0001 complete [ 1070.725366] Lustre: server umount lustre-OST0000 complete [ 1074.650704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.820769] LustreError: 93292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.878393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.241765] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1079.053165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.295726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1080.585914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1082.397933] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1083.610519] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1087.904853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1087.987895] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1087.992450] LustreError: 93280:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1088.002160] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1088.005930] LustreError: 95146:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 1088.008924] LustreError: 95146:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1088.011496] LustreError: 95146:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1088.037775] Lustre: server umount lustre-OST0000 complete [ 1088.039069] LustreError: 95146:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1090.829646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.893772] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1090.897957] LustreError: 93282:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1090.909108] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1090.912168] LustreError: 95614:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 1090.914680] LustreError: 95614:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1090.916686] LustreError: 95614:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1090.936420] Lustre: server umount lustre-OST0000 complete [ 1090.937665] LustreError: 95614:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1093.890996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1093.954680] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1093.956449] Lustre: Found index 0 for lustre-OST0000, updating log [ 1093.958436] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1093.979945] Lustre: lustre-OST0000: new disk, initializing [ 1093.981643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1094.067746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.886930] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1097.111762] Lustre: Failing over lustre-OST0000 [ 1097.160181] Lustre: server umount lustre-OST0000 complete [ 1099.232616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.233483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1099.238898] Lustre: Skipped 1 previous similar message [ 1099.241213] Lustre: Skipped 2 previous similar messages [ 1104.352133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.353811] Lustre: Skipped 1 previous similar message [ 1104.475663] Lustre: server umount lustre-MDT0000 complete [ 1105.722940] LustreError: 95167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478861 with bad export cookie 11306344496284010972 [ 1105.726381] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.727609] LustreError: 95167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1105.841756] Lustre: server umount lustre-MDT0001 complete [ 1108.829449] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:14:24 (1772478864) [ 1113.995292] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1117.363757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.508934] LustreError: 98452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.534528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.805232] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1121.545351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.106467] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1124.004793] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1126.128690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.252260] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1131.032692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1131.067050] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1131.101501] Lustre: lustre-OST0001: new disk, initializing [ 1131.103558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1133.121325] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1136.196859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1136.642279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1136.646264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1136.689292] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1144.900061] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1151.968428] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.968995] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.972418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1151.973261] Lustre: Skipped 1 previous similar message [ 1157.092379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1157.094171] Lustre: Skipped 5 previous similar messages [ 1157.332278] Lustre: server umount lustre-MDT0001 complete [ 1159.483570] LustreError: 98448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.466732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.501407] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1160.504442] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1160.520773] Lustre: 98442:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1160.776341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1160.778260] Lustre: Skipped 3 previous similar messages [ 1162.128243] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1162.149324] LustreError: 102455:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1162.152344] LustreError: 102455:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1162.155384] LustreError: 98444:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1162.187198] LustreError: 102468:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.202.110@tcp: cfg command failed: rc = -17 [ 1162.191681] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1162.191681] [ 1165.198994] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 14:15:21 (1772478921) [ 1165.799409] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1170.555234] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1176.523224] Lustre: 104283:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1184.375589] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1185.457830] Lustre: 105414:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1237.473187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.473757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.473907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1237.477184] Lustre: Skipped 4 previous similar messages [ 1237.484769] Lustre: Skipped 3 previous similar messages [ 1241.507731] Lustre: server umount lustre-MDT0000 complete [ 1242.592645] LustreError: 100893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.597963] LustreError: 100893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1242.767918] LustreError: 98433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478998 with bad export cookie 11306344496284011882 [ 1242.768958] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.771747] LustreError: 98433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1242.898084] Lustre: server umount lustre-MDT0001 complete [ 1254.932187] Lustre: server umount lustre-OST0000 complete [ 1266.476245] Lustre: server umount lustre-OST0001 complete [ 1268.211508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1271.273033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1273.964490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1276.976728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1283.328555] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1286.759962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.765579] Lustre: lustre-MDT0000: reset Object Index mappings [ 1286.918783] LustreError: 109315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.944991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.247889] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1290.885197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.891245] Lustre: lustre-MDT0001: reset Object Index mappings [ 1292.375142] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1293.334254] Lustre: 110430:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1295.479843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1296.743423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1297.508195] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1300.281055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1301.547762] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1301.549047] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1301.549077] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1302.387587] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1305.473721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1306.638020] Lustre: 112280:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1326.174642] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:18:02 (1772479082) [ 1357.796601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.800382] Lustre: Skipped 3 previous similar messages [ 1357.801218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1357.801780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.805063] Lustre: Skipped 3 previous similar messages [ 1361.649260] Lustre: server umount lustre-MDT0000 complete [ 1362.784847] LustreError: 110048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479118 with bad export cookie 11306344496284058656 [ 1362.788331] LustreError: 110048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1362.789397] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.912078] LustreError: 109311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.916246] LustreError: 109311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1377.247080] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1377.344987] Lustre: server umount lustre-MDT0001 complete [ 1384.702991] Lustre: server umount lustre-OST0000 complete [ 1400.287155] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1400.372223] Lustre: server umount lustre-OST0001 complete [ 1402.470317] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 1405.593813] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1417.526549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.618652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1417.629800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1417.668244] Lustre: lustre-MDT0000: new disk, initializing [ 1417.689408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.691145] Lustre: Skipped 3 previous similar messages [ 1417.696327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1418.961414] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1421.892797] Lustre: server umount lustre-MDT0000 complete [ 1425.056031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.157039] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1425.163170] Lustre: 116487:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1425.170639] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1426.556321] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1428.213574] Lustre: server umount lustre-MDT0000 complete [ 1430.861580] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:19:46 (1772479186) [ 1431.366762] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1431.931302] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:19:47 (1772479187) [ 1437.108282] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1440.422250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.900566] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1444.608304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.646652] Lustre: 118499:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1444.671984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1444.673629] Lustre: Skipped 1 previous similar message [ 1444.709944] Lustre: lustre-MDT0001: new disk, initializing [ 1444.758917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1444.762681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1445.983986] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1448.187968] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1450.371110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1450.485291] Lustre: lustre-OST0000: new disk, initializing [ 1450.487167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1451.202696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1451.208379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1451.234511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1452.349880] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1456.388649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1457.537701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1457.541200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1457.574770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1458.336747] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1462.788662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1463.989133] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1614.116667] Lustre: 127253:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1614.119489] Lustre: 127253:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1614.121320] Lustre: 127253:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1614.124251] Lustre: 127253:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1614.126118] Lustre: 127253:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1614.127950] Lustre: 127253:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1683.273810] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:23:58 (1772479438) [ 1695.711864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1695.714317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.717258] Lustre: Skipped 2 previous similar messages [ 1695.718667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.721260] Lustre: Skipped 8 previous similar messages [ 1700.780763] Lustre: server umount lustre-MDT0000 complete [ 1702.185944] LustreError: 121639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479458 with bad export cookie 11306344496284111464 [ 1702.191910] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1702.193253] LustreError: 121639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1703.393139] LustreError: 118504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.404307] LustreError: 118504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1704.413531] Lustre: server umount lustre-MDT0001 complete [ 1712.061793] Lustre: server umount lustre-OST0000 complete [ 1719.790885] Lustre: server umount lustre-OST0001 complete [ 1722.318572] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 1725.350995] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1738.923070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.054324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1739.068944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1739.070922] Lustre: Skipped 1 previous similar message [ 1739.113593] Lustre: lustre-MDT0000: new disk, initializing [ 1739.114958] Lustre: Skipped 1 previous similar message [ 1739.139220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.142075] Lustre: Skipped 5 previous similar messages [ 1739.149085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1740.518365] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1744.853334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.904167] Lustre: 130232:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1745.010614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1746.256216] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1749.321746] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1750.698150] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1752.837612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1754.940966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1754.975876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1755.053962] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1758.133567] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1761.308441] Lustre: lustre-MDD0000: changelog on [ 1762.151678] Lustre: lustre-MDD0001: changelog on [ 2194.846058] Lustre: lustre-MDD0001: changelog off [ 2195.624157] Lustre: 130241:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2195.985732] Lustre: lustre-MDD0000: changelog off [ 2200.544194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2200.547218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.548046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2200.552851] Lustre: Skipped 6 previous similar messages [ 2200.555534] Lustre: Skipped 6 previous similar messages [ 2212.319128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2212.395557] Lustre: server umount lustre-OST0000 complete [ 2220.395723] Lustre: server umount lustre-MDT0000 complete [ 2221.024188] LustreError: 130244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.032560] LustreError: 130244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2222.098052] LustreError: 131011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479978 with bad export cookie 11306344496284875381 [ 2222.100778] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.103141] LustreError: 131011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2222.273235] Lustre: server umount lustre-MDT0001 complete [ 2226.756755] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2228.205631] Key type lgssc unregistered [ 2228.360412] LNet: 140383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.364598] LNetError: 140383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.376807] LNet: Removed LNI 192.168.202.110@tcp [ 2228.711210] Key type .llcrypt unregistered [ 2228.712195] Key type ._llcrypt unregistered [ 2235.101558] Key type ._llcrypt registered [ 2235.102593] Key type .llcrypt registered [ 2235.137088] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2241.237299] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2241.627994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.680044] alg: No test for adler32 (adler32-zlib) [ 2242.543400] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2242.633381] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2244.215137] Key type lgssc registered [ 2244.586172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2255.481095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2255.486071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.579658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2256.592659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2256.633226] Lustre: lustre-MDT0000: new disk, initializing [ 2256.658297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.664869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2257.988931] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2262.249187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.292130] Lustre: 143523:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2262.319648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2262.321896] Lustre: Skipped 1 previous similar message [ 2262.360840] Lustre: lustre-MDT0001: new disk, initializing [ 2262.402474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2262.416142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2262.421356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2263.736570] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2266.836195] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2268.033715] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2270.323836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2270.420817] Lustre: lustre-OST0000: new disk, initializing [ 2270.422795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2270.452977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.370681] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2273.016793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2273.020559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2273.051488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2275.486285] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2277.623535] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_module obdecho/obdecho [ 2278.229975] LustreError: 146139:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2278.232199] LustreError: 146139:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 2278.833347] LustreError: 146236:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2282.975786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2282.977528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.980702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.489074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.490048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.495571] Lustre: Skipped 1 previous similar message [ 2287.963503] Lustre: server umount lustre-MDT0000 complete [ 2289.136476] LustreError: 144300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480045 with bad export cookie 4097636135937069348 [ 2289.138913] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2289.139841] LustreError: 144300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2289.247645] Lustre: server umount lustre-MDT0001 complete [ 2300.829730] Lustre: server umount lustre-OST0000 complete [ 2307.781713] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2310.878567] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2322.557832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.647906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2322.659063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2322.700983] Lustre: lustre-MDT0000: new disk, initializing [ 2322.722294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.729694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2324.028553] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2328.286058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.327000] Lustre: 149909:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2328.346808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2328.348671] Lustre: Skipped 1 previous similar message [ 2328.385919] Lustre: lustre-MDT0001: new disk, initializing [ 2328.422533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.435844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2328.439979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2329.873544] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2333.087074] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.342133] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.658691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2336.779276] Lustre: lustre-OST0000: new disk, initializing [ 2336.781889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2336.811765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.780155] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2341.884603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2341.888907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2341.929650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2341.947952] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2343.465543] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2343.539583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2344.928775] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2345.006113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2346.977894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.980339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2346.982503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.982509] Lustre: Skipped 2 previous similar messages [ 2346.983493] Lustre: Skipped 1 previous similar message [ 2351.763368] Lustre: server umount lustre-OST0000 complete [ 2354.144059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2354.147084] LustreError: Skipped 1 previous similar message [ 2354.148336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.153796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2354.156134] Lustre: Skipped 1 previous similar message [ 2359.261048] Lustre: server umount lustre-MDT0000 complete [ 2360.569155] LustreError: 149902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480116 with bad export cookie 4097636135937070293 [ 2360.572234] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2360.573217] LustreError: 149902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2360.707670] Lustre: server umount lustre-MDT0001 complete [ 2361.937901] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:35:17 (1772480117) [ 2362.491739] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2363.123610] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:35:18 (1772480118) [ 2365.948301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.109662] LustreError: 153912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.115525] LustreError: 153912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2366.131236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.505689] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2370.628118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.249217] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2374.186505] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2375.535405] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2377.917957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2380.105748] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2382.071478] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2401.759917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2401.763247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.767984] Lustre: Skipped 1 previous similar message [ 2401.770472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.772877] Lustre: Skipped 2 previous similar messages [ 2405.463569] Lustre: server umount lustre-MDT0000 complete [ 2406.629862] LustreError: 153894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480162 with bad export cookie 4097636135937071161 [ 2406.632428] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.632952] LustreError: 153894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2407.738719] LustreError: 155524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.935606] LustreError: 154628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.935797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2411.939551] LustreError: 154628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2411.942843] Lustre: Skipped 6 previous similar messages [ 2412.776064] Lustre: server umount lustre-MDT0001 complete [ 2414.832172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2418.817474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.895870] LustreError: 157434:0:(llog.c:1643:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2418.898485] Lustre: 157434:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2428.219072] LustreError: 157439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.339193] LustreError: 157439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.347680] LustreError: 157439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2441.695702] LustreError: 157441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.700032] LustreError: 157441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2458.938850] LustreError: 157460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.943884] LustreError: 157460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2479.711284] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2479.714214] Lustre: 157434:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.202.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2492.385239] LustreError: 157472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.391085] LustreError: 157472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2505.247324] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2505.249650] Lustre: 153905:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.202.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2520.551981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2520.555184] Lustre: Skipped 2 previous similar messages [ 2521.962982] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2524.999849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2525.663345] LustreError: 153905:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 2525.666877] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2527.071208] LustreError: 158161:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2527.074617] LustreError: 158161:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ce7ef4c5 x1858579986478080/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1772480283 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2527.083133] LustreError: 158161:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2527.711558] LustreError: 141748:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005489764d x1858579986482560/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2529.010740] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2530.641351] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2531.158404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2532.221126] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2532.837047] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2535.778040] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2542.052883] Lustre: server umount lustre-MDT0000 complete [ 2543.073535] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.076391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.076789] Lustre: Skipped 4 previous similar messages [ 2543.259613] LustreError: 153895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480299 with bad export cookie 4097636135937077622 [ 2543.262328] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2543.264931] LustreError: 153895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2543.376943] Lustre: server umount lustre-MDT0001 complete [ 2554.835624] Lustre: server umount lustre-OST0000 complete [ 2561.054926] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2563.877511] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2576.723851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.837152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2576.850209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2576.893828] Lustre: lustre-MDT0000: new disk, initializing [ 2576.932450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2578.323071] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2582.619688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.668251] Lustre: 162248:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2582.793879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2584.172560] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2587.299119] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2588.532355] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2590.846629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2590.985052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.988271] Lustre: Skipped 3 previous similar messages [ 2592.952498] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2596.047376] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2597.324973] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2599.934950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2599.978659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2600.452557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2601.887448] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2601.977812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2605.026946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.027420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.028189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.030599] LustreError: Skipped 1 previous similar message [ 2605.035282] Lustre: Skipped 2 previous similar messages [ 2608.794989] Lustre: server umount lustre-OST0000 complete [ 2616.212958] Lustre: server umount lustre-MDT0000 complete [ 2617.494206] LustreError: 162241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480373 with bad export cookie 4097636135937094422 [ 2617.497991] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2617.498225] LustreError: 162241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2617.624272] Lustre: server umount lustre-MDT0001 complete [ 2620.413108] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:39:36 (1772480376) [ 2622.750801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.885569] LustreError: 166272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.892311] LustreError: 166272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 2624.144874] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2626.848727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.197322] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2629.964720] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2631.202276] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2633.246579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2635.048039] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2636.678741] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2656.204628] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:40:12 (1772480412) [ 2658.273375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.273976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.276852] Lustre: Skipped 2 previous similar messages [ 2658.277583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.282012] Lustre: Skipped 6 previous similar messages [ 2663.244952] Lustre: server umount lustre-OST0000 complete [ 2670.550438] Lustre: server umount lustre-MDT0000 complete [ 2671.826292] LustreError: 166254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480427 with bad export cookie 4097636135937095290 [ 2671.829665] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.830284] LustreError: 166254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2671.946344] Lustre: server umount lustre-MDT0001 complete [ 2675.522103] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2676.555069] Key type lgssc unregistered [ 2676.675411] LNet: 169668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2676.677887] LNetError: 169668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2676.686404] LNet: Removed LNI 192.168.202.110@tcp [ 2676.986988] Key type .llcrypt unregistered [ 2676.988242] Key type ._llcrypt unregistered [ 2684.491580] Key type ._llcrypt registered [ 2684.493161] Key type .llcrypt registered [ 2684.538391] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2684.849566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2684.866524] alg: No test for adler32 (adler32-zlib) [ 2685.733374] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2685.819977] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2687.407109] Key type lgssc registered [ 2687.741736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2690.271130] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2690.276668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.695293] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2710.815146] LustreError: 170875:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [ 2715.935347] LustreError: 170875:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2715.940695] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2715.943192] LustreError: MGC192.168.202.110@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2715.949723] LustreError: 170875:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2715.954819] LustreError: 170875:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 2715.957698] LustreError: 170875:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2715.978925] Lustre: server umount lustre-OST0000 complete [ 2715.979984] LustreError: 170875:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2718.510678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.648992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.808141] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2723.489632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.608338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.878406] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2726.745528] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2727.937554] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2730.098885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.165869] LustreError: 172959:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2730.168734] Lustre: 172959:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.202.110@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2730.207824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.997107] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2733.703084] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2738.657929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.661233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.662530] Lustre: Skipped 2 previous similar messages [ 2738.666299] Lustre: Skipped 2 previous similar messages [ 2740.562806] Lustre: server umount lustre-MDT0000 complete [ 2741.690675] LustreError: 171355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480497 with bad export cookie 10443743482540880947 [ 2741.694611] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2741.695738] LustreError: 171355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2741.801957] Lustre: server umount lustre-MDT0001 complete [ 2753.235855] Lustre: server umount lustre-OST0000 complete [ 2755.890250] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2758.976823] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2769.968533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.056022] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2770.065713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2770.096172] Lustre: lustre-MDT0000: new disk, initializing [ 2770.116757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.122230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2771.366918] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2775.461712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.508858] Lustre: 176334:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2775.538716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2775.540488] Lustre: Skipped 1 previous similar message [ 2775.571652] Lustre: lustre-MDT0001: new disk, initializing [ 2775.604224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.615531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2775.618304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2776.856328] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2779.862722] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2781.042322] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2783.211965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2783.328835] Lustre: lustre-OST0000: new disk, initializing [ 2783.331590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2785.181672] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2788.121758] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2789.366743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2789.370827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2789.399398] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2789.404621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2789.471816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2790.713018] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2790.773897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2794.469826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.470165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2794.471091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2794.474036] Lustre: Skipped 1 previous similar message [ 2794.478294] LustreError: Skipped 1 previous similar message [ 2797.466207] Lustre: server umount lustre-OST0000 complete [ 2799.584486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.585147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.590027] Lustre: Skipped 1 previous similar message [ 2799.591706] Lustre: Skipped 1 previous similar message [ 2804.704159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.706308] Lustre: Skipped 2 previous similar messages [ 2809.824397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2809.826488] Lustre: Skipped 1 previous similar message [ 2812.895128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2812.972784] Lustre: server umount lustre-MDT0000 complete [ 2814.306593] LustreError: 178075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480570 with bad export cookie 10443743482540881633 [ 2814.310370] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2814.311799] LustreError: 178075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2814.437617] Lustre: server umount lustre-MDT0001 complete [ 2817.436259] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:42:53 (1772480573) [ 2823.579869] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2824.761419] Key type lgssc unregistered [ 2824.890345] LNet: 180853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.893727] LNetError: 180853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2824.902411] LNet: Removed LNI 192.168.202.110@tcp [ 2825.218295] Key type .llcrypt unregistered [ 2825.219806] Key type ._llcrypt unregistered [ 2831.772816] Key type ._llcrypt registered [ 2831.774374] Key type .llcrypt registered [ 2831.803819] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2832.176686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2832.182254] alg: No test for adler32 (adler32-zlib) [ 2833.023308] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2833.107751] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2834.703186] Key type lgssc registered [ 2835.126961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2837.500473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2837.505532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.645903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.828962] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2842.396762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.526343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.763108] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2845.419105] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2846.558798] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2849.101341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.170692] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2849.171935] LustreError: 183663:0:(llog.c:1684:llog_backup()) MGC192.168.202.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2849.174292] Lustre: 183663:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.202.110@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2849.215529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.106687] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2852.910206] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2853.621915] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.622858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2853.626329] Lustre: Skipped 1 previous similar message [ 2853.631664] Lustre: Skipped 2 previous similar messages [ 2858.976787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.979461] Lustre: Skipped 2 previous similar messages [ 2859.798179] Lustre: server umount lustre-MDT0000 complete [ 2861.016286] LustreError: 182011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480617 with bad export cookie 12846217392159767343 [ 2861.019765] LustreError: 182011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2861.019841] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.130530] Lustre: server umount lustre-MDT0001 complete [ 2872.660552] Lustre: server umount lustre-OST0000 complete [ 2875.524181] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2878.533353] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2890.209196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.294291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2890.305058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2890.340783] Lustre: lustre-MDT0000: new disk, initializing [ 2890.363471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.369686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2891.701830] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2895.774893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.818414] Lustre: 187043:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2895.837446] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2895.839753] Lustre: Skipped 1 previous similar message [ 2895.842032] LustreError: 187808:0:(llog.c:1684:llog_backup()) MGC192.168.202.110@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2895.846045] LustreError: 187808:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 2895.848984] Lustre: 187808:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2895.858070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2895.860713] Lustre: Skipped 1 previous similar message [ 2895.895821] Lustre: lustre-MDT0001: new disk, initializing [ 2895.928895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2895.943498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2895.948991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2897.169704] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2900.171181] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2901.346700] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2903.669619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2903.779223] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2903.781023] Lustre: Skipped 1 previous similar message [ 2903.782700] LustreError: 188759:0:(llog.c:1684:llog_backup()) MGC192.168.202.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2903.785934] LustreError: 188759:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 2903.788081] Lustre: 188759:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2903.791156] Lustre: 188759:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2903.802834] Lustre: lustre-OST0000: new disk, initializing [ 2903.805802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2905.712510] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2908.667485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2908.671820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2908.703224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2908.872777] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2910.174970] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2910.251712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2911.597676] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2911.675293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2913.762330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.762402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2913.766014] Lustre: Skipped 1 previous similar message [ 2913.766798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.768024] LustreError: Skipped 1 previous similar message [ 2918.413914] Lustre: server umount lustre-OST0000 complete [ 2921.439941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2921.442796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.448060] Lustre: Skipped 1 previous similar message [ 2921.451150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.453323] Lustre: Skipped 1 previous similar message [ 2924.001081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.849449] Lustre: server umount lustre-MDT0000 complete [ 2927.042042] LustreError: 187035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480683 with bad export cookie 12846217392159768043 [ 2927.044454] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.045156] LustreError: 187035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2927.157054] Lustre: server umount lustre-MDT0001 complete [ 2929.916596] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:44:45 (1772480685) [ 2934.847508] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2938.101479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.245278] LustreError: 191740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.250625] LustreError: 191740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2938.259585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.261470] Lustre: Skipped 1 previous similar message [ 2939.547200] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2942.320329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.708235] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2944.637119] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2946.780851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2948.778811] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2951.616318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2951.684439] Lustre: lustre-OST0001: new disk, initializing [ 2951.686073] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2953.089039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2953.092409] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2953.125147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2953.665130] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2959.087511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2964.777212] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2970.403488] Lustre: DEBUG MARKER: ADD OST3 [ 2972.101937] loop: module loaded [ 2972.122871] loop0: detected capacity change from 0 to 204800000 [ 2972.239340] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2972.243646] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2972.249259] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2974.276942] loop0: detected capacity change from 0 to 204800000 [ 2976.211247] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.290852] Lustre: lustre-OST0003: new disk, initializing [ 2976.293351] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 2976.339312] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 2976.342129] Lustre: Skipped 3 previous similar messages [ 2977.270047] Lustre: DEBUG MARKER: STOP OST3 [ 2977.949559] Lustre: Failing over lustre-OST0003 [ 2977.983814] Lustre: server umount lustre-OST0003 complete [ 2978.192888] LustreError: 194227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.421260] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2982.391229] Lustre: DEBUG MARKER: START OST3 again [ 2982.726967] LustreError: 194522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.732336] LustreError: 194522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2984.685229] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.771084] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 2986.119275] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2986.126473] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2986.677283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 2986.680809] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 2986.683978] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2986.689766] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 2999.925805] Lustre: Failing over lustre-OST0003 [ 2999.963361] Lustre: server umount lustre-OST0003 complete [ 3000.293433] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.294440] LustreError: 194227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.297398] Lustre: Skipped 1 previous similar message [ 3000.302517] LustreError: 194227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3005.410682] LustreError: 196952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.415441] LustreError: 196952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3014.112397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3014.114905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.119463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.121179] Lustre: Skipped 2 previous similar messages [ 3015.651387] LustreError: 193199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.661658] LustreError: 193199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3018.588384] Lustre: server umount lustre-MDT0000 complete [ 3019.723330] LustreError: 191721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480775 with bad export cookie 12846217392159768911 [ 3019.726308] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3019.727169] LustreError: 191721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3024.800156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3024.801752] Lustre: Skipped 7 previous similar messages [ 3034.079145] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3034.155487] Lustre: server umount lustre-MDT0001 complete [ 3035.375338] Lustre: server umount lustre-OST0000 complete [ 3036.564084] Lustre: server umount lustre-OST0001 complete [ 3038.511306] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 3041.046205] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3052.532145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.623290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3052.634954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3052.669391] Lustre: lustre-MDT0000: new disk, initializing [ 3052.692032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.694232] Lustre: Skipped 1 previous similar message [ 3052.699642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3053.903817] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3057.877027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.919109] Lustre: 201071:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3058.026837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3059.303311] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3062.432277] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3063.615452] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3065.851788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3067.511971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3067.547625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3067.764654] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3070.642137] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3071.932861] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3071.995594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3073.178264] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3073.237431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3078.112906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.113782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.114707] LustreError: Skipped 1 previous similar message [ 3078.117946] Lustre: Skipped 5 previous similar messages [ 3078.118532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.121966] Lustre: Skipped 3 previous similar messages [ 3088.351148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3088.407789] Lustre: server umount lustre-OST0000 complete [ 3095.701194] Lustre: server umount lustre-MDT0000 complete [ 3096.849176] LustreError: 201063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480852 with bad export cookie 12846217392159771151 [ 3096.852342] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3096.855061] LustreError: 201063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3096.960873] Lustre: server umount lustre-MDT0001 complete [ 3098.066699] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:47:33 (1772480853) [ 3102.940634] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 3106.039289] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3118.311559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.418645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3118.430737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3118.433043] Lustre: Skipped 3 previous similar messages [ 3118.479661] Lustre: lustre-MDT0000: new disk, initializing [ 3118.481628] Lustre: Skipped 2 previous similar messages [ 3118.507785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.510204] Lustre: Skipped 2 previous similar messages [ 3118.516673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3118.519759] Lustre: Skipped 2 previous similar messages [ 3119.829131] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3123.922972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.965640] Lustre: 207231:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3124.060810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3125.366184] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3128.371747] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3129.519631] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3131.637676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.567280] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3133.668919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3136.495852] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3137.905182] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3137.993614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3139.399182] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3139.475160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3144.162406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.162417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3144.163486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.166020] Lustre: Skipped 4 previous similar messages [ 3144.168370] Lustre: Skipped 6 previous similar messages [ 3154.399153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3154.455765] Lustre: server umount lustre-OST0000 complete [ 3161.684559] Lustre: server umount lustre-MDT0000 complete [ 3163.016542] LustreError: 208011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480919 with bad export cookie 12846217392159772033 [ 3163.019848] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.022667] LustreError: 208011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3163.150695] Lustre: server umount lustre-MDT0001 complete [ 3166.121293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.259131] LustreError: 211042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.264066] LustreError: 211042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3167.374967] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3169.930814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.249965] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3172.909115] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3174.143597] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3176.637379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.767687] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3180.574336] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3201.504668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3201.506976] LustreError: Skipped 2 previous similar messages [ 3203.470277] Lustre: server umount lustre-OST0000 complete [ 3210.836629] Lustre: server umount lustre-MDT0000 complete [ 3211.744269] LustreError: 211042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.749629] LustreError: 211042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3212.107340] LustreError: 211022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480968 with bad export cookie 12846217392159772915 [ 3212.110989] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.111207] LustreError: 211022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3212.237319] Lustre: server umount lustre-MDT0001 complete [ 3316.287105] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3317.299511] Key type lgssc unregistered [ 3317.418317] LNet: 214370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3317.420383] LNetError: 214370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3317.428284] LNet: Removed LNI 192.168.202.110@tcp [ 3317.735157] Key type .llcrypt unregistered [ 3317.737163] Key type ._llcrypt unregistered [ 3325.934245] Key type ._llcrypt registered [ 3325.935525] Key type .llcrypt registered [ 3325.975856] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 3332.579218] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3333.030529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3333.041635] alg: No test for adler32 (adler32-zlib) [ 3333.889546] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 3333.971941] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3335.551092] Key type lgssc registered [ 3335.913898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3347.694401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3347.699777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.799382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3348.809664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3348.845986] Lustre: lustre-MDT0000: new disk, initializing [ 3348.868472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.876147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3350.136988] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3354.103821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.142565] Lustre: 217661:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3354.162834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3354.164355] Lustre: Skipped 1 previous similar message [ 3354.199973] Lustre: lustre-MDT0001: new disk, initializing [ 3354.230250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3354.241400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3354.244281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3355.465661] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3358.440337] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3359.621769] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3361.787701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.893693] Lustre: lustre-OST0000: new disk, initializing [ 3361.895548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3361.923443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.706324] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3365.047846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3365.051968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3365.082233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3366.679300] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3368.188818] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3368.257718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3369.572514] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3369.652789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3370.466855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3370.468291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.470152] LustreError: Skipped 1 previous similar message [ 3370.475282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.477409] Lustre: Skipped 1 previous similar message [ 3375.585773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3375.587693] Lustre: Skipped 1 previous similar message [ 3376.406678] Lustre: server umount lustre-OST0000 complete [ 3380.192247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3380.194412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.197904] Lustre: Skipped 1 previous similar message [ 3380.199730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.846696] Lustre: server umount lustre-MDT0000 complete [ 3385.129560] LustreError: 217651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481141 with bad export cookie 11298292810808507487 [ 3385.132642] LustreError: 217651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3385.133636] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3385.261480] Lustre: server umount lustre-MDT0001 complete [ 3391.706791] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3395.364420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.550838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.898616] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3399.795773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.927944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.267969] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3402.215367] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3404.421356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.412784] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3409.334870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3409.421247] Lustre: lustre-OST0001: new disk, initializing [ 3409.423494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3409.463531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3409.465563] Lustre: Skipped 1 previous similar message [ 3410.446706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3410.452623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3410.498373] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3411.524228] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3417.088995] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3419.732061] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3428.485082] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:53:04 (1772481184) [ 3430.880351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3430.883465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.886936] Lustre: Skipped 1 previous similar message [ 3430.889107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.890979] Lustre: Skipped 1 previous similar message [ 3435.292496] Lustre: server umount lustre-MDT0000 complete [ 3436.535362] LustreError: 222147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481192 with bad export cookie 11298292810808508355 [ 3436.538563] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.539933] LustreError: 222147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3436.657300] Lustre: server umount lustre-MDT0001 complete [ 3448.194768] Lustre: server umount lustre-OST0000 complete [ 3459.679813] Lustre: server umount lustre-OST0001 complete [ 3461.724376] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 3465.070934] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3477.046591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.160234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3477.170869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3477.211472] Lustre: lustre-MDT0000: new disk, initializing [ 3477.257914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.265113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3478.593199] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3482.746500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.789891] Lustre: 228314:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3482.902633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3484.190456] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3487.202577] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3488.367238] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3490.599502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.710125] Lustre: lustre-OST0000: new disk, initializing [ 3490.712424] Lustre: Skipped 1 previous similar message [ 3490.714595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3490.717689] Lustre: Skipped 2 previous similar messages [ 3492.087851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3492.092283] Lustre: Skipped 1 previous similar message [ 3492.095031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3492.130400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3492.580317] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3495.624411] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3497.002538] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3497.072130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3498.369765] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3498.444323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3502.562057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.562645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3502.566289] Lustre: Skipped 3 previous similar messages [ 3502.567027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3502.569217] LustreError: Skipped 1 previous similar message [ 3502.570672] Lustre: Skipped 4 previous similar messages [ 3512.802277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.804105] Lustre: Skipped 3 previous similar messages [ 3513.311076] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3513.378928] Lustre: server umount lustre-OST0000 complete [ 3517.922361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.930277] Lustre: Skipped 2 previous similar messages [ 3520.812465] Lustre: server umount lustre-MDT0000 complete [ 3522.024309] LustreError: 228307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481278 with bad export cookie 11298292810808510455 [ 3522.027161] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.027827] LustreError: 228307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3522.143782] Lustre: server umount lustre-MDT0001 complete [ 3524.918657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.070124] LustreError: 232123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.075923] LustreError: 232123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3525.093122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.095085] Lustre: Skipped 2 previous similar messages [ 3526.336857] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3529.249358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.776810] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3532.626783] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3533.792822] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3536.141533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3538.184966] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3539.989266] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3682.784044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3682.787458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.792278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3682.794547] Lustre: Skipped 2 previous similar messages [ 3688.201949] Lustre: server umount lustre-OST0000 complete [ 3693.536579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3693.541738] LustreError: Skipped 1 previous similar message [ 3695.778176] Lustre: server umount lustre-MDT0000 complete [ 3697.130497] LustreError: 232103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481453 with bad export cookie 11298292810808511337 [ 3697.132802] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.133771] LustreError: 232103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3697.241493] Lustre: server umount lustre-MDT0001 complete [ 3803.342900] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3804.447852] Key type lgssc unregistered [ 3804.580618] LNet: 235374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3804.583882] LNetError: 235374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3804.593509] LNet: Removed LNI 192.168.202.110@tcp [ 3804.943247] Key type .llcrypt unregistered [ 3804.944841] Key type ._llcrypt unregistered [ 3813.868466] Key type ._llcrypt registered [ 3813.869403] Key type .llcrypt registered [ 3813.904108] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 3820.318564] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3820.634285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3820.711444] alg: No test for adler32 (adler32-zlib) [ 3821.595643] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 3821.695250] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3823.287181] Key type lgssc registered [ 3823.863377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3843.024141] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3846.446541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3846.452222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.546522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3847.557814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3847.602727] Lustre: lustre-MDT0000: new disk, initializing [ 3847.627414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.635603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3848.892273] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3853.097985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.149880] Lustre: 239367:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3853.182879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3853.187393] Lustre: Skipped 1 previous similar message [ 3853.234187] Lustre: lustre-MDT0001: new disk, initializing [ 3853.268600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.281907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3853.285606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3854.536508] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3856.871114] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3859.168959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3859.294581] Lustre: lustre-OST0000: new disk, initializing [ 3859.298217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3859.338967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.189224] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3865.244148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3865.300928] Lustre: lustre-OST0001: new disk, initializing [ 3865.302610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3865.333975] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3865.863093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3865.866090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3865.870190] Lustre: Skipped 1 previous similar message [ 3865.925226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3865.936947] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3867.235943] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3872.920655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3879.832839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3889.120071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3889.123408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.127960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.681670] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.685753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.686148] Lustre: Skipped 2 previous similar messages [ 3891.688198] Lustre: Skipped 2 previous similar messages [ 3895.090687] Lustre: server umount lustre-MDT0000 complete [ 3896.675666] LustreError: 239358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481652 with bad export cookie 15953108490245591648 [ 3896.681837] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3896.684076] LustreError: 239358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3896.820207] Lustre: server umount lustre-MDT0001 complete [ 3908.756206] Lustre: server umount lustre-OST0000 complete [ 3920.772441] Lustre: server umount lustre-OST0001 complete [ 3926.896405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3930.941366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3934.852041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3939.010434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3946.307763] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3950.184756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.192400] Lustre: lustre-MDT0000: reset Object Index mappings [ 3950.392963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.695670] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3954.924926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3954.931189] Lustre: lustre-MDT0001: reset Object Index mappings [ 3956.726454] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3957.969951] Lustre: 247902:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3960.774054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.914803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.916613] Lustre: Skipped 1 previous similar message [ 3963.093416] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3965.030219] LustreError: 248256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.036115] LustreError: 248256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3966.676593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.166613] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3975.790177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 3975.793183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3983.392454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3984.632146] Lustre: 249767:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3991.525624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.528082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3991.533217] Lustre: Skipped 2 previous similar messages [ 3991.535773] Lustre: Skipped 2 previous similar messages [ 3996.129747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3996.135868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.146673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3996.150107] Lustre: Skipped 1 previous similar message [ 3997.301107] Lustre: server umount lustre-MDT0000 complete [ 3998.655571] LustreError: 246779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481754 with bad export cookie 15953108490245598326 [ 3998.659356] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3998.660619] LustreError: 246779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3998.791388] Lustre: server umount lustre-MDT0001 complete [ 4010.000372] Lustre: server umount lustre-OST0000 complete [ 4021.086214] Lustre: server umount lustre-OST0001 complete [ 4027.505456] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:03:03 (1772481783) [ 4033.135154] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4037.180750] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4050.574559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.669680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4050.681651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4050.718988] Lustre: lustre-MDT0000: new disk, initializing [ 4050.744490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4050.748086] Lustre: Skipped 1 previous similar message [ 4050.755384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4050.759325] Lustre: Skipped 1 previous similar message [ 4052.061662] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4056.384433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.424911] Lustre: 253988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4056.524414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4056.528293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4057.780506] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4061.085398] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4062.411971] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4065.072460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4065.183823] Lustre: lustre-OST0000: new disk, initializing [ 4065.185223] Lustre: Skipped 1 previous similar message [ 4065.186847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4065.188963] Lustre: Skipped 2 previous similar messages [ 4066.811636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4066.814180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4066.849286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4067.301003] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4070.574569] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4072.025557] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4072.116281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4073.429954] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4073.503179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4076.238441] Lustre: server umount lustre-OST0000 complete [ 4082.145060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.145490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4082.145852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.145855] Lustre: Skipped 3 previous similar messages [ 4082.148329] Lustre: Skipped 1 previous similar message [ 4087.312609] Lustre: server umount lustre-MDT0000 complete [ 4088.852066] LustreError: 253980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481844 with bad export cookie 15953108490245600419 [ 4088.858434] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4088.861456] LustreError: 253980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4098.101641] Lustre: server umount lustre-MDT0001 complete [ 4105.265189] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4108.641302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.803547] LustreError: 258493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4108.821201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4108.823345] Lustre: Skipped 2 previous similar messages [ 4110.180518] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4113.671476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.056779] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4116.101786] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4118.444336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4120.755271] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4124.693440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4124.795233] Lustre: lustre-OST0001: new disk, initializing [ 4124.798449] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4126.157743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4126.162404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4126.207168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4127.189436] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4131.959463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4133.323728] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4140.166520] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4140.172952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4144.608142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4144.610349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.614188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.616113] Lustre: Skipped 3 previous similar messages [ 4148.157233] Lustre: server umount lustre-MDT0000 complete [ 4149.995450] LustreError: 258476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772481906 with bad export cookie 15953108490245601301 [ 4150.000723] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.119644] Lustre: server umount lustre-MDT0001 complete [ 4160.961988] Lustre: server umount lustre-OST0000 complete [ 4172.570201] Lustre: server umount lustre-OST0001 complete [ 4178.158099] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4182.346740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.524113] LustreError: 263772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.543092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.545143] Lustre: Skipped 3 previous similar messages [ 4183.806404] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4187.518838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.616542] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4187.620399] Lustre: Skipped 4 previous similar messages [ 4189.168752] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4190.462238] Lustre: 264879:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4193.361954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.064453] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4197.669297] LustreError: 265231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4198.834496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.087685] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4207.920902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4207.953503] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4215.252815] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4216.952831] Lustre: 266749:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4223.648930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4223.667097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4227.199760] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:06:23 (1772481983) [ 4238.823988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.825733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4238.826959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4238.827649] Lustre: Skipped 5 previous similar messages [ 4253.151061] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4253.221090] Lustre: server umount lustre-MDT0000 complete [ 4254.181395] LustreError: 266486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.188622] LustreError: 266486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4254.820303] LustreError: 263753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772482010 with bad export cookie 15953108490245607608 [ 4254.825439] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4254.826221] LustreError: 263753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4260.982965] Lustre: server umount lustre-MDT0001 complete [ 4268.348658] Lustre: server umount lustre-OST0000 complete [ 4275.701415] Lustre: server umount lustre-OST0001 complete [ 4278.020446] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4281.378458] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4297.346134] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4301.573351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.700124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4301.714699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4301.766525] Lustre: lustre-MDT0000: new disk, initializing [ 4301.797656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4303.294442] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4307.989895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.030894] Lustre: 270930:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4308.127162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4309.292318] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4313.762062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4313.918918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.921501] Lustre: Skipped 5 previous similar messages [ 4314.866038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4315.805061] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4319.917377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4321.411683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4322.005704] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4326.387099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4336.344213] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 15:08:12 (1772482092) [ 4337.122058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.123479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4337.127159] Lustre: Skipped 5 previous similar messages [ 4337.128604] Lustre: Skipped 15 previous similar messages [ 4343.264428] Lustre: server umount lustre-MDT0000 complete [ 4344.450154] LustreError: 271708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772482100 with bad export cookie 15953108490245615511 [ 4344.453292] LustreError: 271708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4344.453595] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.569368] Lustre: server umount lustre-MDT0001 complete [ 4356.610257] Lustre: server umount lustre-OST0000 complete [ 4367.065561] Lustre: server umount lustre-OST0001 complete [ 4373.188551] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4376.366367] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4389.553575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.648754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4389.651399] Lustre: Skipped 2 previous similar messages [ 4389.659610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4389.661149] Lustre: Skipped 4 previous similar messages [ 4389.697505] Lustre: lustre-MDT0000: new disk, initializing [ 4389.698761] Lustre: Skipped 3 previous similar messages [ 4389.725690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4389.728233] Lustre: Skipped 3 previous similar messages [ 4390.975653] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4395.542724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.593326] Lustre: 277979:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4395.720762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4395.723247] Lustre: Skipped 2 previous similar messages [ 4396.978456] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4400.649726] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4402.166630] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4404.593366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4405.921985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4406.717179] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4410.249780] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4411.282235] Lustre: Mounted lustre-client [ 4411.596100] LustreError: 280474:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff97055f873000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4411.623454] Lustre: Unmounted lustre-client [ 4416.485826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.487322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4416.500676] LustreError: Skipped 1 previous similar message [ 4417.103665] Lustre: server umount lustre-OST0000 complete [ 4424.681829] Lustre: server umount lustre-MDT0000 complete [ 4426.118380] LustreError: 277972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772482182 with bad export cookie 15953108490245617037 [ 4426.122551] LustreError: 277972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4426.122777] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4426.268527] Lustre: server umount lustre-MDT0001 complete [ 4430.507338] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4431.821093] Key type lgssc unregistered [ 4463.711488] LNet: 281753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4463.717373] LNetError: 281753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4543.589515] LNet: Removed LNI 192.168.202.110@tcp [ 4544.046206] Key type .llcrypt unregistered [ 4544.048320] Key type ._llcrypt unregistered [ 4552.430168] Key type ._llcrypt registered [ 4552.431791] Key type .llcrypt registered [ 4552.488618] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4558.861725] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4559.228127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4559.237165] alg: No test for adler32 (adler32-zlib) [ 4560.097297] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 4560.196739] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4561.775129] Key type lgssc registered [ 4562.315052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4575.088438] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:12:10 (1772482330) [ 4577.588218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4577.592938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.693500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4578.713072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4578.763708] Lustre: lustre-MDT0000: new disk, initializing [ 4578.796755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.808184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4580.490977] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4586.028186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.095422] Lustre: 284913:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4586.138818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4586.142420] Lustre: Skipped 1 previous similar message [ 4586.189140] Lustre: lustre-MDT0001: new disk, initializing [ 4586.229475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4586.249413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4586.256329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4587.965885] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4591.941548] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4593.665452] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4596.993130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4597.161830] Lustre: lustre-OST0000: new disk, initializing [ 4597.166005] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4597.214897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.748942] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4603.405936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4603.409548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4603.453250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4603.580130] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4608.484581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.484596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4608.490616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4608.492746] LustreError: Skipped 1 previous similar message [ 4609.997516] Lustre: server umount lustre-OST0000 complete [ 4612.063902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4612.066515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.070698] Lustre: Skipped 1 previous similar message [ 4612.073061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4612.075343] Lustre: Skipped 1 previous similar message [ 4613.235340] Lustre: server umount lustre-MDT0000 complete [ 4613.602394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.956103] LustreError: 285692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772482371 with bad export cookie 3631829349310563607 [ 4614.958779] LustreError: 285692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4614.959757] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4615.061822] Lustre: server umount lustre-MDT0001 complete [ 4618.756923] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4620.058879] Key type lgssc unregistered [ 4620.190364] LNet: 288280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4620.194359] LNetError: 288280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4620.204499] LNet: Removed LNI 192.168.202.110@tcp [ 4620.577096] Key type .llcrypt unregistered [ 4620.579318] Key type ._llcrypt unregistered [ 4631.762700] Key type ._llcrypt registered [ 4631.764189] Key type .llcrypt registered [ 4631.806264] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4632.162339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4632.188057] alg: No test for adler32 (adler32-zlib) [ 4633.034812] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 4633.111288] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4634.687211] Key type lgssc registered [ 4635.250902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4638.689350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4638.701047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.872501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.490781] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4645.518626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.701280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.485713] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4649.587494] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4651.056260] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4653.683418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.859893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.262851] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4658.812637] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4667.193137] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4667.979669] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4670.005151] Lustre: server umount lustre-MDT0000 complete [ 4671.456889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4671.462561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.471836] LustreError: 289658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4671.481540] LustreError: 289658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4671.836104] LustreError: 289639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772482427 with bad export cookie 9284008222363388710 [ 4671.843690] LustreError: 289639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4671.844185] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4674.369271] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [ 4674.381146] LustreError: 289652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.529285] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.537304] Lustre: Skipped 1 previous similar message [ 4676.576385] LustreError: 290814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.576750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4676.586689] LustreError: 290814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4676.594379] Lustre: Skipped 1 previous similar message [ 4678.103373] Lustre: server umount lustre-MDT0001 complete [ 4684.619422] Lustre: lustre-OST0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 4686.039491] Lustre: server umount lustre-OST0000 complete [ 4690.095501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.724306] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4692.912764] Lustre: 293383:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.110@tcp [ 4693.782348] Lustre: server umount MGS complete [ 4697.640855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.826335] LustreError: 293954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.860924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.520744] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4703.202984] LustreError: 293955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.738157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.940463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4705.520245] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4709.658881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4711.953918] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4713.567664] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4726.373878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec [ 4727.384716] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4727.847609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4732.385650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4732.385904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.391783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4732.394594] Lustre: Skipped 1 previous similar message [ 4734.981736] Lustre: server umount lustre-OST0000 complete [ 4737.504840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4737.509548] Lustre: Skipped 1 previous similar message [ 4737.511571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4737.513112] Lustre: Skipped 1 previous similar message [ 4742.225990] Lustre: server umount lustre-MDT0000 complete [ 4742.623887] LustreError: 293955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4742.630110] LustreError: 293955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4743.309071] LustreError: 293936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772482499 with bad export cookie 9284008222363389529 [ 4743.311910] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4743.312028] LustreError: 293936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4743.425743] Lustre: server umount lustre-MDT0001 complete [ 4746.572382] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4747.507711] Key type lgssc unregistered [ 4747.699575] LNet: 297149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4747.704455] LNetError: 297149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4747.714619] LNet: Removed LNI 192.168.202.110@tcp [ 4748.175712] Key type .llcrypt unregistered [ 4748.176941] Key type ._llcrypt unregistered [ 4756.078436] Key type ._llcrypt registered [ 4756.079846] Key type .llcrypt registered [ 4756.128595] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4763.837770] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4764.146413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4764.156727] alg: No test for adler32 (adler32-zlib) [ 4765.032051] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 4765.135176] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4766.719193] Key type lgssc registered [ 4767.204689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4782.175324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4782.185161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.289902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4783.301796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4783.346518] Lustre: lustre-MDT0000: new disk, initializing [ 4783.371567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.379531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4784.903801] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4790.318341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.383522] Lustre: 300245:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4790.421249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4790.424570] Lustre: Skipped 1 previous similar message [ 4790.472143] Lustre: lustre-MDT0001: new disk, initializing [ 4790.513961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4790.534727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4790.540384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4792.136579] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4795.812215] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4797.089489] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4799.642959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4799.770088] Lustre: lustre-OST0000: new disk, initializing [ 4799.772560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4799.806490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.401852] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4805.145660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4805.151843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4805.211502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4806.256301] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4807.924104] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4808.031963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4809.781525] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4809.905413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4815.334791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4815.334942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.339826] LustreError: Skipped 1 previous similar message [ 4815.340776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4815.347789] Lustre: Skipped 1 previous similar message [ 4815.356872] Lustre: Skipped 1 previous similar message [ 4816.969968] Lustre: server umount lustre-OST0000 complete [ 4820.450214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.451927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4820.461162] Lustre: Skipped 1 previous similar message [ 4820.467785] Lustre: Skipped 1 previous similar message [ 4825.569183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4825.573043] Lustre: Skipped 1 previous similar message [ 4830.689172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4830.692912] Lustre: Skipped 1 previous similar message [ 4833.247161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4833.348163] Lustre: server umount lustre-MDT0000 complete [ 4835.177169] LustreError: 300237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772482591 with bad export cookie 14445985171652003374 [ 4835.180811] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.182027] LustreError: 300237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4835.292684] Lustre: server umount lustre-MDT0001 complete [ 4839.391833] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 15:16:35 (1772482595) [ 4861.402239] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 15:16:57 (1772482617) [ 4878.727348] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4879.868926] Key type lgssc unregistered [ 4879.982334] LNet: 306788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4879.984399] LNetError: 306788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4879.992256] LNet: Removed LNI 192.168.202.110@tcp [ 4880.278323] Key type .llcrypt unregistered [ 4880.280554] Key type ._llcrypt unregistered [ 4914.777913] Key type ._llcrypt registered [ 4914.779110] Key type .llcrypt registered [ 4914.825771] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4914.979476] Key type .llcrypt unregistered [ 4914.981412] Key type ._llcrypt unregistered [ 4929.027640] Key type ._llcrypt registered [ 4929.029063] Key type .llcrypt registered [ 4929.091971] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4929.254358] Key type .llcrypt unregistered [ 4929.256491] Key type ._llcrypt unregistered [ 4942.917293] Key type ._llcrypt registered [ 4942.918924] Key type .llcrypt registered [ 4942.959431] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4943.095452] Key type .llcrypt unregistered [ 4943.097346] Key type ._llcrypt unregistered [ 4981.227668] Key type ._llcrypt registered [ 4981.229370] Key type .llcrypt registered [ 4981.290361] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4981.458472] Key type .llcrypt unregistered [ 4981.460598] Key type ._llcrypt unregistered [ 4994.574876] Key type ._llcrypt registered [ 4994.576507] Key type .llcrypt registered [ 4994.634614] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4994.796291] Key type .llcrypt unregistered [ 4994.798461] Key type ._llcrypt unregistered [ 5008.786986] Key type ._llcrypt registered [ 5008.788448] Key type .llcrypt registered [ 5008.849868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5009.010433] Key type .llcrypt unregistered [ 5009.012660] Key type ._llcrypt unregistered [ 5022.511961] Key type ._llcrypt registered [ 5022.513217] Key type .llcrypt registered [ 5022.560237] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5022.709696] Key type .llcrypt unregistered [ 5022.711756] Key type ._llcrypt unregistered [ 5036.419783] Key type ._llcrypt registered [ 5036.420851] Key type .llcrypt registered [ 5036.462392] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5036.590296] Key type .llcrypt unregistered [ 5036.592430] Key type ._llcrypt unregistered [ 5050.334154] Key type ._llcrypt registered [ 5050.335565] Key type .llcrypt registered [ 5050.374418] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5050.510420] Key type .llcrypt unregistered [ 5050.511860] Key type ._llcrypt unregistered [ 5079.103104] Key type ._llcrypt registered [ 5079.104643] Key type .llcrypt registered [ 5079.164855] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5079.329332] Key type .llcrypt unregistered [ 5079.331437] Key type ._llcrypt unregistered [ 5093.222281] Key type ._llcrypt registered [ 5093.223590] Key type .llcrypt registered [ 5093.277526] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5093.430603] Key type .llcrypt unregistered [ 5093.433059] Key type ._llcrypt unregistered [ 5107.216299] Key type ._llcrypt registered [ 5107.218038] Key type .llcrypt registered [ 5107.254969] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5107.350184] Key type .llcrypt unregistered [ 5107.351990] Key type ._llcrypt unregistered [ 5132.814451] Key type ._llcrypt registered [ 5132.816285] Key type .llcrypt registered [ 5132.863625] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5140.990737] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5141.302741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5141.311941] alg: No test for adler32 (adler32-zlib) [ 5142.204587] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5142.314841] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 5143.903220] Key type lgssc registered [ 5144.542725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5159.760304] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5050 sec ========== 15:21:55 (1772482915) [ 5160.534326] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:21:56 (1772482916) === [ 5162.000924] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:21:57 (1772482917) === [ 5179.320291] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5180.751265] Key type lgssc unregistered [ 5180.921626] LNet: 325601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5180.926271] LNetError: 325601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5180.940507] LNet: Removed LNI 192.168.202.110@tcp [ 5181.373833] Key type .llcrypt unregistered [ 5181.376201] Key type ._llcrypt unregistered