[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 599646270 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003090] x2apic enabled [ 0.005007] Switched APIC routing to physical x2apic. [ 0.006023] kvm-guest: setup PV IPIs [ 0.010000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011074] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012020] pid_max: default: 32768 minimum: 301 [ 0.014167] LSM: Security Framework initializing [ 0.015066] Yama: becoming mindful. [ 0.016069] SELinux: Initializing. [ 0.017097] *** VALIDATE selinux *** [ 0.032677] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.041581] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.042256] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.043130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.044152] *** VALIDATE tmpfs *** [ 0.046102] *** VALIDATE proc *** [ 0.047363] *** VALIDATE cgroup *** [ 0.049017] *** VALIDATE cgroup2 *** [ 0.051147] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.052507] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.053017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.054043] Spectre V2 : User space: Vulnerable [ 0.055015] Speculative Store Bypass: Vulnerable [ 0.059855] debug: unmapping init [mem 0xffffffff9f059000-0xffffffff9f060fff] [ 0.062656] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.064495] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.065045] ... version: 2 [ 0.066029] ... bit width: 48 [ 0.067022] ... generic registers: 4 [ 0.068728] ... value mask: 0000ffffffffffff [ 0.069196] ... max period: 00007fffffffffff [ 0.070027] ... fixed-purpose events: 3 [ 0.071028] ... event mask: 000000070000000f [ 0.073442] rcu: Hierarchical SRCU implementation. [ 0.077258] smp: Bringing up secondary CPUs ... [ 0.079056] x86: Booting SMP configuration: [ 0.080303] .... node #0, CPUs: #1 #2 #3 [ 0.093017] smp: Brought up 1 node, 4 CPUs [ 0.095016] smpboot: Max logical packages: 1 [ 0.096015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.274628] node 0 deferred pages initialised in 176ms [ 0.280223] devtmpfs: initialized [ 0.282333] x86/mm: Memory block size: 128MB [ 0.286669] gcov: version magic: 0x41383552 [ 0.291165] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.295102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.301296] pinctrl core: initialized pinctrl subsystem [ 0.303151] [ 0.304008] ************************************************************* [ 0.307014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.311021] ** ** [ 0.313091] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.316014] ** ** [ 0.318013] ** This means that this kernel is built to expose internal ** [ 0.320011] ** IOMMU data structures, which may compromise security on ** [ 0.322011] ** your system. ** [ 0.324011] ** ** [ 0.326018] ** If you see this message and you are not debugging the ** [ 0.329013] ** kernel, report this immediately to your vendor! ** [ 0.332011] ** ** [ 0.334010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.336012] ************************************************************* [ 0.339052] NET: Registered protocol family 16 [ 0.342716] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.344044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.345035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.348017] cpuidle: using governor menu [ 0.349684] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.352450] PCI: Using configuration type 1 for base access [ 0.353133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.359109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.360019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.361481] cryptd: max_cpu_qlen set to 1000 [ 0.362242] ACPI: Added _OSI(Module Device) [ 0.363049] ACPI: Added _OSI(Processor Device) [ 0.365029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.368013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.374047] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.382555] ACPI: Interpreter enabled [ 0.384094] ACPI: PM: (supports S0 S3 S4 S5) [ 0.385013] ACPI: Using IOAPIC for interrupt routing [ 0.388268] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.391705] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.405280] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.408039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.411028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.415087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.420802] acpiphp: Slot [2] registered [ 0.423112] acpiphp: Slot [5] registered [ 0.425119] acpiphp: Slot [6] registered [ 0.426115] acpiphp: Slot [7] registered [ 0.428309] acpiphp: Slot [8] registered [ 0.430119] acpiphp: Slot [9] registered [ 0.431182] acpiphp: Slot [10] registered [ 0.433144] acpiphp: Slot [3] registered [ 0.435102] acpiphp: Slot [4] registered [ 0.437166] acpiphp: Slot [11] registered [ 0.439175] acpiphp: Slot [12] registered [ 0.441200] acpiphp: Slot [13] registered [ 0.443188] acpiphp: Slot [14] registered [ 0.445177] acpiphp: Slot [15] registered [ 0.447140] acpiphp: Slot [16] registered [ 0.449175] acpiphp: Slot [17] registered [ 0.451258] acpiphp: Slot [18] registered [ 0.453133] acpiphp: Slot [19] registered [ 0.455115] acpiphp: Slot [20] registered [ 0.456277] acpiphp: Slot [21] registered [ 0.459127] acpiphp: Slot [22] registered [ 0.462136] acpiphp: Slot [23] registered [ 0.464108] acpiphp: Slot [24] registered [ 0.466132] acpiphp: Slot [25] registered [ 0.467122] acpiphp: Slot [26] registered [ 0.469141] acpiphp: Slot [27] registered [ 0.471122] acpiphp: Slot [28] registered [ 0.473099] acpiphp: Slot [29] registered [ 0.475112] acpiphp: Slot [30] registered [ 0.477133] acpiphp: Slot [31] registered [ 0.479066] PCI host bridge to bus 0000:00 [ 0.481021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.483022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.486023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.489027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.492193] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.496025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.498269] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.502153] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.505271] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.517016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.523051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.525038] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.528018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.532019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.535596] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.537995] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.541038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.544879] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.548832] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.560015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.566007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.573272] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.588017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.594018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.628016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.640913] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.649017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.718020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.874016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.888000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.902018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.911018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.936018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.948991] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.959019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.972024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.002018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.021268] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.033020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.046018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.079017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.094753] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.103015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.110030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.138016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.160566] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.162682] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.166437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.169393] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.172245] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.179088] iommu: Default domain type: Passthrough [ 1.181469] SCSI subsystem initialized [ 1.184161] ACPI: bus type USB registered [ 1.186103] usbcore: registered new interface driver usbfs [ 1.188111] usbcore: registered new interface driver hub [ 1.190085] usbcore: registered new device driver usb [ 1.193191] pps_core: LinuxPPS API ver. 1 registered [ 1.195010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.199105] PTP clock support registered [ 1.202058] EDAC MC: Ver: 3.0.0 [ 1.205073] PCI: Using ACPI for IRQ routing [ 1.208575] NetLabel: Initializing [ 1.210010] NetLabel: domain hash size = 128 [ 1.213018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.216084] NetLabel: unlabeled traffic allowed by default [ 1.219510] vgaarb: loaded [ 1.221815] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.224012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.231177] clocksource: Switched to clocksource kvm-clock [ 1.410358] VFS: Disk quotas dquot_6.6.0 [ 1.412206] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.414847] *** VALIDATE ramfs *** [ 1.416247] *** VALIDATE hugetlbfs *** [ 1.418275] pnp: PnP ACPI init [ 1.421153] pnp: PnP ACPI: found 6 devices [ 1.441874] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.445039] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.447304] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.449976] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.452448] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.455358] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.459241] NET: Registered protocol family 2 [ 1.462048] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.468083] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.472483] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.479445] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.483568] TCP: Hash tables configured (established 65536 bind 65536) [ 1.487455] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.491960] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.498540] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.503269] NET: Registered protocol family 1 [ 1.506604] RPC: Registered named UNIX socket transport module. [ 1.509305] RPC: Registered udp transport module. [ 1.511492] RPC: Registered tcp transport module. [ 1.513357] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.516469] NET: Registered protocol family 44 [ 1.518100] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.520912] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.523519] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.526434] PCI: CLS 0 bytes, default 64 [ 1.529251] Unpacking initramfs... [ 4.055312] debug: unmapping init [mem 0xffff8f673cc54000-0xffff8f673ffbffff] [ 4.060309] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.063131] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.067060] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.897710] Initialise system trusted keyrings [ 4.898996] Key type blacklist registered [ 4.902069] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.914500] zbud: loaded [ 4.918303] *** VALIDATE nfs *** [ 4.919434] *** VALIDATE nfs4 *** [ 4.921287] pstore: using deflate compression [ 4.925516] Platform Keyring initialized [ 5.073361] NET: Registered protocol family 38 [ 5.077111] Key type asymmetric registered [ 5.078870] Asymmetric key parser 'x509' registered [ 5.081121] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.084426] io scheduler mq-deadline registered [ 5.086530] io scheduler kyber registered [ 5.088389] io scheduler bfq registered [ 5.090570] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.093819] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.096566] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.099442] ACPI: Power Button [PWRF] [ 5.105096] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.112402] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.139904] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.156580] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.199252] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.236216] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.280557] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.287752] Non-volatile memory driver v1.3 [ 5.290353] Linux agpgart interface v0.103 [ 5.354513] virtio_blk virtio1: [vda] 134128 512-byte logical blocks (68.7 MB/65.5 MiB) [ 5.360883] vda: detected capacity change from 0 to 68673536 [ 5.406250] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.420778] vdb: detected capacity change from 0 to 1073741824 [ 5.447163] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.451837] vdc: detected capacity change from 0 to 2621440000 [ 5.617459] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.621103] vdd: detected capacity change from 0 to 2621440000 [ 5.668334] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.672988] vde: detected capacity change from 0 to 4294967296 [ 5.704989] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.713314] vdf: detected capacity change from 0 to 4294967296 [ 5.729792] libphy: Fixed MDIO Bus: probed [ 5.736949] usbcore: registered new interface driver usbserial_generic [ 5.748788] usbserial: USB Serial support registered for generic [ 5.752860] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.765359] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.768380] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.771792] mousedev: PS/2 mouse device common for all mice [ 5.773950] rtc_cmos 00:05: RTC can wake from S4 [ 5.783657] rtc_cmos 00:05: registered as rtc0 [ 5.784638] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.786503] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.786543] intel_pstate: CPU model not supported [ 5.792469] hid: raw HID events driver (C) Jiri Kosina [ 5.801807] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.803289] usbcore: registered new interface driver usbhid [ 5.826709] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.839972] usbhid: USB HID core driver [ 5.842458] drop_monitor: Initializing network drop monitor service [ 5.872343] Initializing XFRM netlink socket [ 5.874452] NET: Registered protocol family 10 [ 5.877701] Segment Routing with IPv6 [ 5.879205] NET: Registered protocol family 17 [ 5.881291] mpls_gso: MPLS GSO support [ 5.887359] RAS: Correctable Errors collector initialized. [ 5.889530] AVX version of gcm_enc/dec engaged. [ 5.891860] AES CTR mode by8 optimization enabled [ 6.002282] sched_clock: Marking stable (6002190803, 0)->(7733951701, -1731760898) [ 6.006313] registered taskstats version 1 [ 6.008453] Loading compiled-in X.509 certificates [ 6.012141] zswap: loaded using pool lzo/zbud [ 6.063991] Key type big_key registered [ 6.087244] Key type encrypted registered [ 6.090352] ima: No TPM chip found, activating TPM-bypass! [ 6.094169] ima: Allocated hash algorithm: sha1 [ 6.100167] ima: No architecture policies found [ 6.102190] evm: Initialising EVM extended attributes: [ 6.104315] evm: security.selinux [ 6.106701] evm: security.ima [ 6.112563] evm: security.capability [ 6.114318] evm: HMAC attrs: 0x1 [ 6.117340] rtc_cmos 00:05: setting system clock to 2026-04-11 00:09:17 UTC (1775866157) [ 6.134232] debug: unmapping init [mem 0xffffffffa0003000-0xffffffffa01fffff] [ 6.141196] debug: unmapping init [mem 0xffffffff9ed82000-0xffffffff9f058fff] [ 6.152211] Write protecting the kernel read-only data: 28672k [ 6.157973] debug: unmapping init [mem 0xffffffff9d403000-0xffffffff9d5fffff] [ 6.161117] debug: unmapping init [mem 0xffffffff9dd14000-0xffffffff9ddfffff] [ 6.198453] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.211139] systemd[1]: Detected virtualization kvm. [ 6.216259] systemd[1]: Detected architecture x86-64. [ 6.220120] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.265980] systemd[1]: No hostname configured. [ 6.270555] systemd[1]: Set hostname to . [ 6.276321] random: systemd: uninitialized urandom read (16 bytes read) [ 6.281154] systemd[1]: Initializing machine ID from random generator. [ 6.492755] random: ln: uninitialized urandom read (6 bytes read) [ 6.648757] random: systemd: uninitialized urandom read (16 bytes read) [ 6.651790] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.656913] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.663149] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.823631] device-mapper: uevent: version 1.0.3 [ 7.825269] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target S[ 9.255746] virtio_net virtio0 ens2: renamed from eth0 ystem Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.528464] random: fast init done [ 10.014388] scsi host0: ata_piix [ 10.376841] scsi host1: ata_piix [ 10.379052] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.382182] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.781845] random: crng init done [ 15.784941] random: 7 urandom warning(s) missed due to ratelimiting [ 17.200210] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.122901] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.558699] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.349853] SELinux: Disabled at runtime. [ 22.413350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.423069] systemd[1]: Detected virtualization kvm. [ 22.425160] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.254369] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.260756] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.267254] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.273045] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.276829] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.288210] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.323213] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ 24.432269] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 25.708147] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.670906] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.721325] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.690852] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.798136] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [[ 33.015094] Key type dns_resolver registered *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.765186] NFS: Registering the id_resolver key type [ 33.767739] Key type id_resolver registered [ 33.770224] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg214-server login: [ 69.923941] libcfs: loading out-of-tree module taints kernel. [ 69.943916] Key type ._llcrypt registered [ 69.945838] Key type .llcrypt registered [ 70.007160] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 79.610086] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 80.350141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.359832] alg: No test for adler32 (adler32-zlib) [ 81.405976] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 81.854196] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 83.519961] Key type lgssc registered [ 84.390988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.208563] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.982344] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 121.299118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 121.327881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 122.464096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 122.485178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 122.538510] Lustre: lustre-MDT0000: new disk, initializing [ 122.596798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 122.608670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 124.455792] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.480240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.526161] Lustre: 6472:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 131.548012] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 131.552670] Lustre: Skipped 1 previous similar message [ 131.599287] Lustre: lustre-MDT0001: new disk, initializing [ 131.654994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 131.677984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 131.683122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 133.359980] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.931361] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 140.216744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 140.319744] Lustre: lustre-OST0000: new disk, initializing [ 140.324727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 140.356151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.999493] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.397200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 145.401081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 145.434454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 149.462877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 149.510397] Lustre: lustre-OST0001: new disk, initializing [ 149.512619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 149.548455] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.352097] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.853982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 152.859672] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 152.910878] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 160.830378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 163.768466] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 171.459244] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing check_logdir /tmp/testlogs/ [ 174.601964] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing yml_node [ 181.136328] Lustre: DEBUG MARKER: Client: 2.17.52.3 [ 184.832785] Lustre: DEBUG MARKER: MDS: 2.17.52.3 [ 188.492696] Lustre: DEBUG MARKER: OSS: 2.17.52.3 [ 191.505177] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 10 20:12:19 EDT 2026 [ 196.589988] hrtimer: interrupt took 15993215 ns [ 214.906967] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 217.252853] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 234.977338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.992372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 235.026304] Lustre: Skipped 1 previous similar message [ 240.097648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 240.103094] Lustre: Skipped 6 previous similar messages [ 241.086490] Lustre: server umount lustre-MDT0000 complete [ 244.736527] LustreError: 6466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775866396 with bad export cookie 7562249796925570892 [ 244.741761] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 244.744217] LustreError: 6466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 245.219919] LustreError: 6479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 245.220374] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.224342] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 245.252827] LustreError: 6479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 245.276702] Lustre: Skipped 3 previous similar messages [ 249.248575] LustreError: 6481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 249.249520] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 249.278410] Lustre: Skipped 2 previous similar messages [ 254.432046] LustreError: 6479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 254.432266] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 254.442925] LustreError: 6479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 254.461881] Lustre: Skipped 1 previous similar message [ 259.039615] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 259.271508] Lustre: server umount lustre-MDT0001 complete [ 262.799667] Lustre: server umount lustre-OST0000 complete [ 266.138816] Lustre: server umount lustre-OST0001 complete [ 271.457289] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 278.611450] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 306.698613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 306.944969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 306.972989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 307.078818] Lustre: lustre-MDT0000: new disk, initializing [ 307.164084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.180567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 310.592407] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 319.326462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 319.414489] Lustre: 14941:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 319.464105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 319.467061] Lustre: Skipped 1 previous similar message [ 319.524247] Lustre: lustre-MDT0001: new disk, initializing [ 319.598596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 319.627957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 319.638248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 324.062209] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 330.742206] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 334.068360] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 339.841524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 340.102768] Lustre: lustre-OST0000: new disk, initializing [ 340.109767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 340.184868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 342.141536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 342.161645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 342.248421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 344.859596] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 351.427657] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 355.350557] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 355.564567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 359.404925] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 359.572262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 362.982840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 362.986701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.993297] LustreError: Skipped 1 previous similar message [ 362.995272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 363.002118] Lustre: Skipped 1 previous similar message [ 363.011483] Lustre: Skipped 3 previous similar messages [ 367.408699] Lustre: server umount lustre-OST0000 complete [ 370.657275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 370.665231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.849684] Lustre: server umount lustre-MDT0000 complete [ 378.337122] LustreError: 14954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.357359] LustreError: 14954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 379.826806] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775866531 with bad export cookie 7562249796925572425 [ 379.830618] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.838047] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 380.168687] Lustre: server umount lustre-MDT0001 complete [ 384.739526] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:15:34 (1775866534) [ 398.707715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.154671] LustreError: 19412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.218628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.936420] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 409.569318] LustreError: 19413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.578350] LustreError: 19413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 409.819828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.801263] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 419.627679] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 423.099820] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 424.814068] Lustre: Failing over lustre-MDT0000 [ 424.960824] Lustre: server umount lustre-MDT0000 complete [ 425.442045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 425.454438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.461558] Lustre: Skipped 1 previous similar message [ 429.537890] LustreError: 20124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.554663] LustreError: 20124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 442.683767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.792892] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.393451] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 448.498652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 448.503850] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 448.518065] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 452.223919] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 458.723598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 458.726128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.746180] Lustre: Skipped 1 previous similar message [ 458.750434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 458.753759] Lustre: Skipped 3 previous similar messages [ 461.573283] Lustre: server umount lustre-MDT0000 complete [ 464.280358] LustreError: 19394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775866615 with bad export cookie 7562249796925573839 [ 464.281649] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.289808] LustreError: 19394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 464.552557] Lustre: server umount lustre-MDT0001 complete [ 471.492936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.057562] LustreError: 22405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.076590] LustreError: 22405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 472.203578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.206286] Lustre: Skipped 2 previous similar messages [ 475.737523] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 483.403087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.657757] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 492.902727] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 496.824982] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 499.082525] Lustre: Failing over lustre-MDT0000 [ 499.169173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.172374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.192842] Lustre: Skipped 1 previous similar message [ 499.202574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 499.204362] Lustre: Skipped 2 previous similar messages [ 499.375551] Lustre: server umount lustre-MDT0000 complete [ 517.105133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.261910] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.724459] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 522.725857] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 522.741511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 522.748514] Lustre: Skipped 1 previous similar message [ 522.761995] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 528.695082] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 532.960407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.967826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.986690] Lustre: Skipped 1 previous similar message [ 538.850651] Lustre: server umount lustre-MDT0000 complete [ 539.105165] LustreError: 23425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.121777] LustreError: 23425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 542.205065] LustreError: 22387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775866693 with bad export cookie 7562249796925574777 [ 542.206087] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.227401] LustreError: 22387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 542.633776] Lustre: server umount lustre-MDT0001 complete [ 549.838636] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:18:19 (1775866699) [ 563.802587] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 572.206095] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 605.136990] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 613.247197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.431600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 613.461953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 613.558123] Lustre: lustre-MDT0000: new disk, initializing [ 613.610381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.615472] Lustre: Skipped 2 previous similar messages [ 613.625392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 617.483961] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 626.377121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.477856] Lustre: 28393:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 626.674843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 630.083881] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 636.052506] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 638.945090] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 644.265312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.741667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 645.784938] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 645.791231] Lustre: Skipped 1 previous similar message [ 645.825443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 645.840791] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 648.985600] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 654.612440] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 658.199499] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 676.838833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.844538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 676.855019] Lustre: Skipped 1 previous similar message [ 676.872383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 676.878516] Lustre: Skipped 4 previous similar messages [ 679.592608] Lustre: server umount lustre-OST0000 complete [ 688.625769] Lustre: server umount lustre-MDT0000 complete [ 691.147802] LustreError: 28384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775866842 with bad export cookie 7562249796925575176 [ 691.151295] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 691.161071] LustreError: 28384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 691.485693] Lustre: server umount lustre-MDT0001 complete [ 698.539712] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 700.704916] Key type lgssc unregistered [ 700.946835] LNet: 31976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.953909] LNetError: 31976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.966314] LNet: Removed LNI 192.168.202.114@tcp [ 701.651145] Key type .llcrypt unregistered [ 701.653457] Key type ._llcrypt unregistered [ 719.738654] Key type ._llcrypt registered [ 719.740346] Key type .llcrypt registered [ 719.813312] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 720.561670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.581946] alg: No test for adler32 (adler32-zlib) [ 721.632524] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 721.861706] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 723.623671] Key type lgssc registered [ 724.541756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 730.178159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 730.207218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.532620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.122593] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 740.837041] LustreError: 33350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.867325] LustreError: 33350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 742.223521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.501356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 745.908718] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 750.438446] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 753.281857] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 758.216939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 758.596448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 763.525327] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 766.822885] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 767.658434] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 781.296510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 787.109315] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:22:17 (1775866937) [ 788.683917] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 789.938338] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 795.120682] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:22:24 (1775866944) [ 801.996213] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:22:31 (1775866951) [ 802.892717] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 809.817719] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 809.824945] Lustre: Skipped 2 previous similar messages [ 811.083196] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:22:41 (1775866961) [ 813.827765] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 813.829899] Lustre: Skipped 3 previous similar messages [ 827.598187] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 827.602489] Lustre: Skipped 6 previous similar messages [ 829.587623] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:22:59 (1775866979) [ 831.523173] Lustre: *** cfs_fail_loc=131b, val=2*** [ 834.345635] Lustre: *** cfs_fail_loc=131b, val=2*** [ 840.941354] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:23:10 (1775866990) [ 844.715308] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 844.719744] Lustre: Skipped 5 previous similar messages [ 851.683543] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:23:21 (1775867001) [ 863.203882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 863.206041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.210970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 863.216529] Lustre: Skipped 1 previous similar message [ 863.224739] Lustre: Skipped 1 previous similar message [ 867.714388] Lustre: server umount lustre-OST0000 complete [ 870.369107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 870.381758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.392945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.443405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.443480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.468076] Lustre: Skipped 1 previous similar message [ 876.568391] Lustre: server umount lustre-MDT0000 complete [ 878.560945] LustreError: 33350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.572436] LustreError: 33350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 879.111366] LustreError: 33330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775867030 with bad export cookie 10440091120956069140 [ 879.120795] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.122973] LustreError: 33330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 879.351681] Lustre: server umount lustre-MDT0001 complete [ 886.017381] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 887.956208] Key type lgssc unregistered [ 888.167470] LNet: 40868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.176407] LNetError: 40868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.191590] LNet: Removed LNI 192.168.202.114@tcp [ 888.869927] Key type .llcrypt unregistered [ 888.871945] Key type ._llcrypt unregistered [ 901.739954] Key type ._llcrypt registered [ 901.744640] Key type .llcrypt registered [ 901.837722] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 912.733550] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 913.565525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 913.592082] alg: No test for adler32 (adler32-zlib) [ 914.546806] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 914.689541] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 916.319170] Key type lgssc registered [ 917.104723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 943.666746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 943.694617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.890509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 944.914890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 944.969371] Lustre: lustre-MDT0000: new disk, initializing [ 945.053239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.075584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 948.183763] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 956.748308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.840502] Lustre: 43966:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 956.889944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 956.893477] Lustre: Skipped 1 previous similar message [ 956.895844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 956.947960] Lustre: lustre-MDT0001: new disk, initializing [ 957.040381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.087028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 957.097816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 961.020248] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 967.672855] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 970.533448] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 976.257859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 976.487697] Lustre: lustre-OST0000: new disk, initializing [ 976.491117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 976.581858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.641505] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 985.194539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 985.208757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 985.295760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 987.489613] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 990.869558] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 991.007571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 994.529136] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 994.680430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1000.929296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1000.930789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.943843] LustreError: Skipped 1 previous similar message [ 1000.974766] Lustre: Skipped 1 previous similar message [ 1000.980832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1000.989303] Lustre: Skipped 1 previous similar message [ 1002.332340] Lustre: server umount lustre-OST0000 complete [ 1006.058954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.076453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1006.082149] Lustre: Skipped 1 previous similar message [ 1011.168067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1011.173373] Lustre: Skipped 1 previous similar message [ 1013.636989] Lustre: server umount lustre-MDT0000 complete [ 1016.502367] LustreError: 45705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775867167 with bad export cookie 11208151953588615466 [ 1016.504355] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.508453] LustreError: 45705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1016.992327] Lustre: server umount lustre-MDT0001 complete [ 1024.618990] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:26:14 (1775867174) [ 1030.220930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.482752] LustreError: 48016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.493179] LustreError: 48016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1030.534398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.946525] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1035.745163] LustreError: 48017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.841064] LustreError: 48016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.261380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.470314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1043.629068] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1048.081816] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1051.033802] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1056.324463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1056.581495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.305492] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1066.107857] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1080.271603] LustreError: 50229:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1081.025506] Lustre: Modifying parameter general.timeout=1 in log params [ 1082.555243] Lustre: Modifying parameter general.timeout=3 in log params [ 1082.562535] Lustre: Skipped 1 previous similar message [ 1084.779373] Lustre: Modifying parameter general.timeout=6 in log params [ 1084.782912] Lustre: Skipped 2 previous similar messages [ 1088.796860] Lustre: Modifying parameter general.timeout=12 in log params [ 1088.806061] Lustre: Skipped 5 previous similar messages [ 1097.043696] Lustre: Modifying parameter general.timeout=25 in log params [ 1097.046320] Lustre: Skipped 12 previous similar messages [ 1113.341705] Lustre: Modifying parameter general.timeout=49 in log params [ 1113.345139] Lustre: Skipped 23 previous similar messages [ 1145.880467] Lustre: Modifying parameter general.timeout=101 in log params [ 1145.882896] Lustre: Skipped 51 previous similar messages [ 1202.026525] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:29:11 (1775867351) [ 1205.672323] Lustre: 57167:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1207.872863] Lustre: 57270:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1210.818955] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1210.828642] Lustre: Skipped 33 previous similar messages [ 1212.097717] Lustre: 57474:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1212.128202] Lustre: 57474:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1216.565465] Lustre: 57684:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1216.571300] Lustre: 57684:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1244.882808] Lustre: 59042:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1244.910237] LustreError: 59042:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1249.989744] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:29:58 (1775867398) [ 1266.145773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.156515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1266.157059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.167400] Lustre: Skipped 2 previous similar messages [ 1269.742773] Lustre: server umount lustre-MDT0000 complete [ 1271.264646] LustreError: 48727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.301890] LustreError: 48727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1273.542945] LustreError: 47998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775867424 with bad export cookie 11208151953588616348 [ 1273.544977] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.555575] LustreError: 47998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1273.973313] Lustre: server umount lustre-MDT0001 complete [ 1288.595623] Lustre: server umount lustre-OST0000 complete [ 1316.356174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.673750] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1316.973531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.777824] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1331.230330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.362354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1331.389793] Lustre: 61574:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1336.432684] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1341.866878] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1345.686891] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1351.711844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1351.936882] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1352.204816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.208619] Lustre: Skipped 1 previous similar message [ 1358.064499] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1363.512239] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1365.390757] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1365.408556] Lustre: Skipped 18 previous similar messages [ 1365.455451] Lustre: 63801:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1374.688626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1374.688806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.692615] LustreError: Skipped 1 previous similar message [ 1374.693270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.693274] Lustre: Skipped 2 previous similar messages [ 1374.725062] Lustre: Skipped 2 previous similar messages [ 1378.751402] Lustre: server umount lustre-OST0000 complete [ 1382.880069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.885437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.936294] Lustre: server umount lustre-MDT0000 complete [ 1390.050955] LustreError: 61584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.074279] LustreError: 61584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1392.664408] LustreError: 61566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775867544 with bad export cookie 11208151953588619309 [ 1392.666467] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.675727] LustreError: 61566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1392.962182] Lustre: server umount lustre-MDT0001 complete [ 1403.382136] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1405.836558] Key type lgssc unregistered [ 1406.218272] LNet: 64995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.227095] LNetError: 64995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.253995] LNet: Removed LNI 192.168.202.114@tcp [ 1407.114712] Key type .llcrypt unregistered [ 1407.119126] Key type ._llcrypt unregistered [ 1437.347806] Key type ._llcrypt registered [ 1437.349810] Key type .llcrypt registered [ 1437.446061] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1439.114677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1439.167267] alg: No test for adler32 (adler32-zlib) [ 1440.268217] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 1440.467463] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1442.103320] Key type lgssc registered [ 1443.501344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1452.124752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1452.143036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.744748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.582416] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1464.301489] LustreError: 66394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.308103] LustreError: 66394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1468.235749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.131495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1475.534951] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1483.246914] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1488.864352] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1497.570177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1498.220448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.848833] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1512.356176] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1523.159434] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1535.458591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.466739] Lustre: Skipped 1 previous similar message [ 1535.471992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1535.489577] Lustre: Skipped 2 previous similar messages [ 1538.147178] Lustre: server umount lustre-MDT0000 complete [ 1540.583017] LustreError: 66391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.632778] LustreError: 66391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1543.077530] LustreError: 66376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775867694 with bad export cookie 5480266791328575824 [ 1543.090856] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.093885] LustreError: 66376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1543.638088] Lustre: server umount lustre-MDT0001 complete [ 1558.565933] Lustre: server umount lustre-OST0000 complete [ 1588.523290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.887529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1588.958307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1589.280948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.476904] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1604.384255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.525433] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1604.560711] Lustre: 70933:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1605.086088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1610.239431] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1616.848545] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1621.800205] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1629.743631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.054048] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1630.504833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.122284] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1645.118426] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1647.315232] Lustre: 73167:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1649.263451] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1656.292856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1656.300293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.323201] Lustre: Skipped 1 previous similar message [ 1656.331767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.801070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1661.132865] Lustre: server umount lustre-OST0000 complete [ 1666.530448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1666.541465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.587681] Lustre: Skipped 1 previous similar message [ 1666.592236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1666.598303] Lustre: Skipped 1 previous similar message [ 1671.834275] Lustre: server umount lustre-MDT0000 complete [ 1672.180776] LustreError: 70940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1672.210457] LustreError: 70940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1676.201077] LustreError: 70924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775867827 with bad export cookie 5480266791328576664 [ 1676.208232] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.217322] LustreError: 70924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1676.701802] Lustre: server umount lustre-MDT0001 complete [ 1688.751912] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1691.405716] Key type lgssc unregistered [ 1691.697534] LNet: 74363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.704804] LNetError: 74363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.716184] LNet: Removed LNI 192.168.202.114@tcp [ 1692.705506] Key type .llcrypt unregistered [ 1692.708044] Key type ._llcrypt unregistered [ 1722.586137] Key type ._llcrypt registered [ 1722.591969] Key type .llcrypt registered [ 1722.756928] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1724.850679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1725.012086] alg: No test for adler32 (adler32-zlib) [ 1726.184878] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 1726.568164] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1728.343902] Key type lgssc registered [ 1730.345957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1739.481660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1739.525596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.341724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.263307] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1751.526233] LustreError: 75762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.542590] LustreError: 75762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1756.439760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.648424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1757.014115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1761.564247] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1767.900979] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1772.133910] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1779.229982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1779.945196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.807301] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1793.457780] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1800.973270] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1801.968946] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1803.845350] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1803.854563] Lustre: Skipped 1 previous similar message [ 1806.077349] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1806.082535] Lustre: Skipped 1 previous similar message [ 1810.308610] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1810.313913] Lustre: Skipped 3 previous similar messages [ 1818.759221] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1818.762318] Lustre: Skipped 8 previous similar messages [ 1835.301417] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1835.313253] Lustre: Skipped 15 previous similar messages [ 1867.919374] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1867.928164] Lustre: Skipped 24 previous similar messages [ 1928.919158] Lustre: 83539:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1932.920226] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:41:21 (1775868081) [ 1934.381775] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1936.581555] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:41:25 (1775868085) [ 1943.022304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.028592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.051169] Lustre: Skipped 1 previous similar message [ 1943.065451] Lustre: Skipped 1 previous similar message [ 1948.130261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.139824] Lustre: Skipped 1 previous similar message [ 1948.749964] Lustre: server umount lustre-OST0000 complete [ 1956.322719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.333654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.366868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.371251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.488586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.498377] Lustre: Skipped 3 previous similar messages [ 1968.095254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1968.613104] LustreError: 75762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1968.631026] LustreError: 75762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1970.265904] Lustre: server umount lustre-MDT0000 complete [ 1972.714232] LustreError: 78533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.739405] LustreError: 78533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1974.056235] LustreError: 75744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775868125 with bad export cookie 14452193150877229758 [ 1974.058144] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.084201] LustreError: 75744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1974.397047] Lustre: server umount lustre-MDT0001 complete [ 1987.655499] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1991.161634] Key type lgssc unregistered [ 1991.593943] LNet: 84777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.635176] LNetError: 84777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.677121] LNet: Removed LNI 192.168.202.114@tcp [ 1992.593306] Key type .llcrypt unregistered [ 1992.594966] Key type ._llcrypt unregistered [ 1999.174482] Key type ._llcrypt registered [ 1999.176657] Key type .llcrypt registered [ 1999.257467] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2005.742719] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules [ 2007.302881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.435457] alg: No test for adler32 (adler32-zlib) [ 2008.738291] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 2009.115383] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2010.880356] Key type lgssc registered [ 2011.856024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2013.484812] LustreError: 85995:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 2036.116017] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2036.150580] LustreError: 85995:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2048.215680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2048.272053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.997581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.861596] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2060.259488] LustreError: 87003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.272325] LustreError: 87003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2065.400687] LustreError: 87002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.060421] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:43:35 (1775868215) [ 2070.500691] LustreError: 87003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.254682] LustreError: 87034:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2071.269737] LustreError: 87034:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2071.276765] LustreError: 87034:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2071.701032] Lustre: server umount lustre-MDT0000 complete [ 2086.074156] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2090.140518] Key type lgssc unregistered [ 2090.640502] LNet: 88495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.647710] LNetError: 88495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.663927] LNet: Removed LNI 192.168.202.114@tcp [ 2091.529547] Key type .llcrypt unregistered [ 2091.531400] Key type ._llcrypt unregistered [ 2113.001961] Key type ._llcrypt registered [ 2113.006941] Key type .llcrypt registered [ 2113.108408] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2114.314949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2114.337988] alg: No test for adler32 (adler32-zlib) [ 2115.504693] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 2115.781229] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2117.463192] Key type lgssc registered [ 2118.770462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2128.364450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2128.409462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.006541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.863054] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2140.128079] LustreError: 89698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.148920] LustreError: 89698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2143.941041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.432085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2149.775111] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2157.754933] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2162.616452] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2169.556373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2170.520584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.436379] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2183.168777] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2234.791393] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:46:24 (1775868384) [ 2236.808524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2239.225736] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:46:28 (1775868388) [ 2245.602568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.603803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.617927] Lustre: Skipped 2 previous similar messages [ 2245.623252] Lustre: Skipped 2 previous similar messages [ 2250.413501] Lustre: server umount lustre-MDT0000 complete [ 2250.732153] LustreError: 89697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.749492] LustreError: 89697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.688618] LustreError: 89678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775868406 with bad export cookie 11121117951195547906 [ 2254.691939] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.703197] LustreError: 89678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2255.393427] Lustre: server umount lustre-MDT0001 complete [ 2271.466048] Lustre: server umount lustre-OST0000 complete [ 2285.784834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.556039] LustreError: 93637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.800740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.994993] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2292.195362] LustreError: 93638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.291551] LustreError: 93637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.888894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.162585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2308.159532] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2314.118891] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2318.590750] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.091341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2336.429158] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2336.441809] LustreError: 93624:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2336.518720] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2336.526765] LustreError: 95496:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 2336.538037] LustreError: 95496:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2336.544675] LustreError: 95496:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2336.686739] Lustre: server umount lustre-OST0000 complete [ 2336.692692] LustreError: 95496:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2346.847902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2347.017331] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2347.023738] LustreError: 93626:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2347.061252] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2347.075518] LustreError: 95963:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 2347.088818] LustreError: 95963:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2347.096779] LustreError: 95963:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2347.184383] Lustre: server umount lustre-OST0000 complete [ 2347.190450] LustreError: 95963:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2358.470120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2358.751179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2358.758128] Lustre: Found index 0 for lustre-OST0000, updating log [ 2358.782113] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2358.885996] Lustre: lustre-OST0000: new disk, initializing [ 2358.890649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2359.156506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.808246] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2369.900723] Lustre: Failing over lustre-OST0000 [ 2370.142740] Lustre: server umount lustre-OST0000 complete [ 2374.527309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.537566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.240970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2379.249879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.279662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.937603] Lustre: server umount lustre-MDT0000 complete [ 2381.808042] LustreError: 94352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.840468] LustreError: 94352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2385.722793] LustreError: 93618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775868537 with bad export cookie 11121117951195550650 [ 2385.728715] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2385.731508] LustreError: 93618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2386.179633] Lustre: server umount lustre-MDT0001 complete [ 2395.498914] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:49:04 (1775868544) [ 2416.107858] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2428.461187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.905790] LustreError: 98814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.087337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.375520] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2443.915128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.918428] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2454.818968] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2460.933986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2461.680903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.689086] Lustre: Skipped 1 previous similar message [ 2467.646958] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2478.996110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.056663] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2479.152586] Lustre: lustre-OST0001: new disk, initializing [ 2479.161960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2485.891940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2485.904208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2486.163119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2489.482046] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2499.178700] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2511.584620] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2523.659271] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [ 2523.676147] Lustre: Skipped 1 previous similar message [ 2527.200250] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2527.205615] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.250053] Lustre: Skipped 2 previous similar messages [ 2528.716490] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [ 2528.727628] Lustre: Skipped 3 previous similar messages [ 2528.865683] Lustre: server umount lustre-MDT0001 complete [ 2532.352443] LustreError: 98815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.366592] LustreError: 98815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2540.527339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.710922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2540.720760] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2540.768253] Lustre: 98802:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2541.647255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2541.658138] Lustre: Skipped 1 previous similar message [ 2542.110760] LustreError: 102611:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2542.118526] LustreError: 102611:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2542.133544] LustreError: 98806:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2542.341310] LustreError: 102619:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.114@tcp: cfg command failed: rc = -17 [ 2542.361267] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2542.361267] [ 2546.665802] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2547.306025] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2558.794117] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 20:51:48 (1775868708) [ 2585.218919] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2606.013274] Lustre: 104680:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2633.626448] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2638.010582] Lustre: 105819:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2730.464523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2730.474925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.485347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.394702] Lustre: server umount lustre-MDT0000 complete [ 2741.239924] LustreError: 98810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.282548] LustreError: 98810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2742.422566] LustreError: 98795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775868893 with bad export cookie 11121117951195551644 [ 2742.425277] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.430118] LustreError: 98795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2742.667469] Lustre: server umount lustre-MDT0001 complete [ 2757.217485] Lustre: server umount lustre-OST0000 complete [ 2771.833715] Lustre: server umount lustre-OST0001 complete [ 2778.028059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2788.460085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2800.220410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2815.038805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2839.717795] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2853.288251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.327383] Lustre: lustre-MDT0000: reset Object Index mappings [ 2854.147275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.551728] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2869.734602] LustreError: 109756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.763566] LustreError: 109756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2870.892517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.924263] Lustre: lustre-MDT0001: reset Object Index mappings [ 2876.218481] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2879.382090] Lustre: 110871:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2888.257374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.699040] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2897.188314] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2899.767333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2908.074367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.168473] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2910.195547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2916.758457] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2926.707613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2931.631654] Lustre: 112724:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2994.076436] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:59:03 (1775869143) [ 3020.258084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.267659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.279194] Lustre: Skipped 3 previous similar messages [ 3020.286758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.293579] Lustre: Skipped 7 previous similar messages [ 3026.165252] Lustre: server umount lustre-MDT0000 complete [ 3029.989481] LustreError: 109739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775869181 with bad export cookie 11121117951195598551 [ 3029.998361] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3036.529834] Lustre: server umount lustre-MDT0001 complete [ 3047.180097] Lustre: server umount lustre-OST0000 complete [ 3051.573234] LustreError: 113685:0:(ldlm_resource.c:1170:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x71:0x0].0x0 (ffff8f668b316500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3058.019355] Lustre: server umount lustre-OST0001 complete [ 3065.402612] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 3078.606987] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3117.616842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.879787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3117.944936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3118.049228] Lustre: lustre-MDT0000: new disk, initializing [ 3118.217021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.225797] Lustre: Skipped 3 previous similar messages [ 3118.253982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3122.225575] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3130.106859] Lustre: server umount lustre-MDT0000 complete [ 3140.206330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.670378] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3140.692219] Lustre: 116955:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3140.700371] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3145.233512] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3150.111933] Lustre: server umount lustre-MDT0000 complete [ 3158.117563] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:01:47 (1775869307) [ 3159.429321] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3161.477208] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:01:50 (1775869310) [ 3180.592354] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3191.693344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.767127] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3204.883840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.040847] Lustre: 118971:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3205.082224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3205.085332] Lustre: Skipped 1 previous similar message [ 3205.139922] Lustre: lustre-MDT0001: new disk, initializing [ 3205.273877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3205.289881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3210.406569] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3215.184306] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3221.678079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.952806] Lustre: lustre-OST0000: new disk, initializing [ 3221.955145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3223.239931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3223.250097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3223.341963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3227.129704] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3236.264059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.466892] Lustre: lustre-OST0001: new disk, initializing [ 3236.471250] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3237.628580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3237.641040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3237.720660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3241.850493] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3250.423722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3253.756893] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3642.541807] Lustre: 127760:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3642.551244] Lustre: 127760:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3642.557671] Lustre: 127760:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3642.565163] Lustre: 127760:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3642.571466] Lustre: 127760:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3642.583195] Lustre: 127760:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3956.055231] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:15:03 (1775870103) [ 3973.091029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3973.098975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.112990] Lustre: Skipped 5 previous similar messages [ 3973.122046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3973.143944] Lustre: Skipped 6 previous similar messages [ 3975.138274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3975.167022] Lustre: Skipped 2 previous similar messages [ 3977.961819] Lustre: server umount lustre-MDT0000 complete [ 3980.263869] LustreError: 122162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.316680] LustreError: 122162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3982.747804] LustreError: 121368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775870134 with bad export cookie 11121117951195667508 [ 3982.759714] LustreError: 121368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3982.761326] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3985.382135] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.398598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3985.412464] Lustre: Skipped 1 previous similar message [ 3985.418836] Lustre: Skipped 4 previous similar messages [ 3989.730498] Lustre: server umount lustre-MDT0001 complete [ 4000.834667] Lustre: server umount lustre-OST0000 complete [ 4011.397788] Lustre: server umount lustre-OST0001 complete [ 4020.411268] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 4033.665608] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4079.400483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.745413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4079.777317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4079.846410] Lustre: lustre-MDT0000: new disk, initializing [ 4079.922916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.930578] Lustre: Skipped 5 previous similar messages [ 4079.945521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4085.793682] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4100.253136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.441510] Lustre: 130787:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4100.511905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4100.515948] Lustre: Skipped 1 previous similar message [ 4100.635250] Lustre: lustre-MDT0001: new disk, initializing [ 4100.823275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4100.832080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4106.665107] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4116.108395] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4121.007472] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4129.275513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.516120] Lustre: lustre-OST0000: new disk, initializing [ 4129.528277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4130.948203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4130.967908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4131.156198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4137.231840] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4145.975483] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4158.273157] Lustre: lustre-MDD0000: changelog on [ 4161.097200] Lustre: lustre-MDD0001: changelog on [ 6147.410560] Lustre: lustre-MDD0001: changelog off [ 6149.714620] Lustre: 130793:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6150.901307] Lustre: lustre-MDD0000: changelog off [ 6163.941264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6163.942995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.949719] LustreError: Skipped 1 previous similar message [ 6163.950461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.968779] Lustre: Skipped 1 previous similar message [ 6163.987020] Lustre: Skipped 4 previous similar messages [ 6165.443566] Lustre: server umount lustre-OST0000 complete [ 6174.185631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.219804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.237771] Lustre: Skipped 1 previous similar message [ 6177.218329] Lustre: server umount lustre-MDT0000 complete [ 6179.305210] LustreError: 130796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.336129] LustreError: 130796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6181.564476] LustreError: 130780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775872332 with bad export cookie 11121117951196484079 [ 6181.573681] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6181.580513] LustreError: 130780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6182.017858] Lustre: server umount lustre-MDT0001 complete [ 6196.404917] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 6199.818628] Key type lgssc unregistered [ 6200.300929] LNet: 141552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6200.317113] LNetError: 141552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6200.348982] LNet: Removed LNI 192.168.202.114@tcp [ 6201.420305] Key type .llcrypt unregistered [ 6201.423856] Key type ._llcrypt unregistered [ 6227.003780] Key type ._llcrypt registered [ 6227.008492] Key type .llcrypt registered [ 6227.189911] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6244.724345] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6246.318149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6246.378435] alg: No test for adler32 (adler32-zlib) [ 6247.568691] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 6247.846783] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 6249.559293] Key type lgssc registered [ 6251.009948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6291.197645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6291.249632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.521328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6292.552555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6292.634488] Lustre: lustre-MDT0000: new disk, initializing [ 6292.774866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6292.799422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6297.201719] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6308.076277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6308.221664] Lustre: 144725:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6308.285520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6308.289940] Lustre: Skipped 1 previous similar message [ 6308.371867] Lustre: lustre-MDT0001: new disk, initializing [ 6308.446753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6308.496321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6308.504708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6313.542435] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6322.897856] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6328.561585] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.759959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.184836] Lustre: lustre-OST0000: new disk, initializing [ 6337.200569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6337.351425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.297494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6344.333954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6344.473034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6345.326315] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6353.537255] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6359.009879] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_module obdecho/obdecho [ 6360.780493] LustreError: 147342:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6360.790611] LustreError: 147342:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 6362.654296] LustreError: 147438:0:(echo_client.c:1881:echo_md_handler()) Can not resolve the path : rc = -2 [ 6374.881114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6374.890797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.938067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6375.409111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.717951] Lustre: server umount lustre-MDT0000 complete [ 6385.008823] LustreError: 144716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775872536 with bad export cookie 3783395642309105943 [ 6385.010046] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6385.025778] LustreError: 144716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6385.504318] Lustre: server umount lustre-MDT0001 complete [ 6400.291771] Lustre: server umount lustre-OST0000 complete [ 6425.487958] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6437.206670] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6480.717551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.019540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6481.054762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6481.152295] Lustre: lustre-MDT0000: new disk, initializing [ 6481.249459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.265467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6486.291629] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6497.451819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.594687] Lustre: 151124:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6497.648381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6497.654086] Lustre: Skipped 1 previous similar message [ 6497.811514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6497.818691] Lustre: Skipped 3 previous similar messages [ 6497.846977] Lustre: lustre-MDT0001: new disk, initializing [ 6497.950657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6497.997400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6498.016701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6503.330418] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6512.235713] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6516.054612] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6523.074736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.318613] Lustre: lustre-OST0000: new disk, initializing [ 6523.323386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6523.435902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6529.166276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6529.180102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6529.396438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6530.630790] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6539.078543] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6544.165267] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6544.392621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6549.668218] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6549.942070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6555.113131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6555.118914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.127865] LustreError: Skipped 1 previous similar message [ 6555.128928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6555.149373] Lustre: Skipped 2 previous similar messages [ 6560.237830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6560.250817] Lustre: Skipped 2 previous similar messages [ 6565.355440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6565.362518] Lustre: Skipped 1 previous similar message [ 6566.367359] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6566.578490] Lustre: server umount lustre-OST0000 complete [ 6575.074388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6575.081200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.112548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6576.977178] Lustre: server umount lustre-MDT0000 complete [ 6580.706840] LustreError: 151135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.728883] LustreError: 151135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6581.014834] LustreError: 151115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775872732 with bad export cookie 3783395642309106902 [ 6581.024986] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6581.351535] Lustre: server umount lustre-MDT0001 complete [ 6585.211526] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:58:54 (1775872734) [ 6587.475687] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6589.150550] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:58:58 (1775872738) [ 6598.408270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.736915] LustreError: 155131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6598.793302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.037333] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6604.257881] LustreError: 155132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.382023] LustreError: 155132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.925925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.381092] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6627.432342] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6632.630817] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6640.147523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6640.815181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6640.817739] Lustre: Skipped 1 previous similar message [ 6647.299368] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6653.813288] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6669.813259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.829612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6669.842058] Lustre: Skipped 2 previous similar messages [ 6669.861596] Lustre: Skipped 3 previous similar messages [ 6674.286517] Lustre: server umount lustre-MDT0000 complete [ 6674.912354] LustreError: 155132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.949605] LustreError: 155132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6678.326700] LustreError: 155112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775872829 with bad export cookie 3783395642309107784 [ 6678.331584] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.331965] LustreError: 155112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6680.033257] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.045379] Lustre: Skipped 1 previous similar message [ 6683.043170] LustreError: 155405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.073485] LustreError: 155405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6684.832122] Lustre: server umount lustre-MDT0001 complete [ 6692.613952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6706.323355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.581874] LustreError: 158659:0:(llog.c:1646:llog_backup()) MGC192.168.202.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6706.591864] Lustre: 158659:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6707.680540] LustreError: 158664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.430172] LustreError: 158664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.450606] LustreError: 158664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6767.584665] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6767.587456] Lustre: 158659:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6806.969161] LustreError: 158664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.008870] LustreError: 158664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 6808.576713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6813.675103] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6814.640285] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6824.890621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.247207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6848.501584] LustreError: 142945:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c9bc5314 x1862137176788480/t0(0) o250->MGC192.168.202.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6848.567965] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6848.587518] Lustre: MGS: Client 63b31754-d8a9-47b1-bede-ea3887cd4368 (at 0@lo) reconnecting [ 6848.598571] Lustre: MGC192.168.202.114@tcp: Connection restored to 0@lo (at 0@lo) [ 6854.161528] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6856.357387] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6863.218623] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6865.153884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6868.487474] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6870.303640] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6892.418517] Lustre: server umount lustre-MDT0000 complete [ 6895.073753] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.076583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6896.648677] LustreError: 155111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775873048 with bad export cookie 3783395642309114273 [ 6896.650020] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6896.662505] LustreError: 155111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6897.272219] Lustre: server umount lustre-MDT0001 complete [ 6911.018405] Lustre: server umount lustre-OST0000 complete [ 6937.733269] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6951.515147] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6995.504293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.013594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6996.080420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6996.235483] Lustre: lustre-MDT0000: new disk, initializing [ 6996.352720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.358382] Lustre: Skipped 1 previous similar message [ 6996.391148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7001.647738] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7012.828916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.975068] Lustre: 163521:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7013.243836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7017.289674] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7024.568903] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7028.992069] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7036.068968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7036.344603] Lustre: lustre-OST0000: new disk, initializing [ 7036.346988] Lustre: Skipped 1 previous similar message [ 7036.351917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7036.355578] Lustre: Skipped 2 previous similar messages [ 7044.248084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7044.255106] Lustre: Skipped 1 previous similar message [ 7044.265987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7044.402274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7044.897157] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7053.752588] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7059.028750] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7059.235788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7064.405710] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7064.572969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7070.179829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.188193] Lustre: Skipped 3 previous similar messages [ 7070.188314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7070.189642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7070.189648] Lustre: Skipped 4 previous similar messages [ 7072.652616] Lustre: server umount lustre-OST0000 complete [ 7083.854721] Lustre: server umount lustre-MDT0000 complete [ 7085.548952] LustreError: 163533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.561539] LustreError: 163533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 7088.274495] LustreError: 164302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775873239 with bad export cookie 3783395642309131087 [ 7088.281223] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7088.292510] LustreError: 164302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7089.088974] Lustre: server umount lustre-MDT0001 complete [ 7099.011751] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 22:07:28 (1775873248) [ 7108.293612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.171260] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7125.684681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.915425] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7137.780669] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7142.661935] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7150.730520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7158.191540] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7165.398323] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7176.661700] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 22:08:45 (1775873325) [ 7183.329055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7183.339621] LustreError: Skipped 1 previous similar message [ 7183.350793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7183.369506] Lustre: Skipped 2 previous similar messages [ 7183.388553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7183.392450] Lustre: Skipped 4 previous similar messages [ 7187.564251] Lustre: server umount lustre-OST0000 complete [ 7197.984833] Lustre: server umount lustre-MDT0000 complete [ 7202.360381] LustreError: 167510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775873353 with bad export cookie 3783395642309131969 [ 7202.371099] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7202.378563] LustreError: 167510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7202.852614] Lustre: server umount lustre-MDT0001 complete [ 7215.319934] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 7218.437585] Key type lgssc unregistered [ 7218.758554] LNet: 170903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7218.774264] LNetError: 170903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7219.817403] LNet: Removed LNI 192.168.202.114@tcp [ 7221.129721] Key type .llcrypt unregistered [ 7221.143646] Key type ._llcrypt unregistered [ 7244.788037] Key type ._llcrypt registered [ 7244.790521] Key type .llcrypt registered [ 7244.972227] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7246.554421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7246.603756] alg: No test for adler32 (adler32-zlib) [ 7247.937994] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 7248.186097] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 7249.856292] Key type lgssc registered [ 7251.449487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7261.207796] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7261.227342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.895460] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7282.016461] LustreError: 172131:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 4: rc = -110 [ 7287.135507] LustreError: 172131:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7287.146565] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7287.160409] LustreError: MGC192.168.202.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7287.172445] LustreError: 172131:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7287.186705] LustreError: 172131:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 7287.199625] LustreError: 172131:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 7287.345363] Lustre: server umount lustre-OST0000 complete [ 7287.347374] LustreError: 172131:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7297.952397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.530240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.569243] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7308.771108] LustreError: 172633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.790538] LustreError: 172633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7313.890584] LustreError: 172632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.408941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.170103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7321.894814] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7329.253966] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7333.629845] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7342.193216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7342.432508] LustreError: 174224:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7342.441201] Lustre: 174224:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.202.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7342.605730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7349.976944] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7356.741217] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7361.021265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7361.028671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7361.049782] Lustre: Skipped 2 previous similar messages [ 7361.068632] Lustre: Skipped 2 previous similar messages [ 7362.084244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.102653] Lustre: Skipped 1 previous similar message [ 7365.687623] Lustre: server umount lustre-MDT0000 complete [ 7367.154458] LustreError: 173558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.167406] LustreError: 173558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7370.957494] LustreError: 173349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775873522 with bad export cookie 17071489124639378871 [ 7370.981295] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.984609] LustreError: 173349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7372.258981] LustreError: 172627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.259380] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.299435] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7372.302167] Lustre: Skipped 1 previous similar message [ 7372.383474] LustreError: 173344:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000036aba248 x1862138225260544/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7372.645402] Lustre: server umount lustre-MDT0001 complete [ 7377.780432] Lustre: server umount lustre-OST0000 complete [ 7388.991630] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7401.248925] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7444.401397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7444.634639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7444.657035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7444.706819] Lustre: lustre-MDT0000: new disk, initializing [ 7444.765259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7444.783390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7450.161949] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7461.977144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7462.140145] Lustre: 177603:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7462.209773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7462.215819] Lustre: Skipped 1 previous similar message [ 7462.317831] Lustre: lustre-MDT0001: new disk, initializing [ 7462.485299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7462.558652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7462.579469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7467.443934] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7475.955485] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7480.234844] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7487.727168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7488.075367] Lustre: lustre-OST0000: new disk, initializing [ 7488.085095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7488.198422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.730098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7493.744453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7493.849537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7494.922972] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7503.384792] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7508.314870] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7508.591995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7514.517694] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7514.917460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7519.712446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7519.717128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.720617] LustreError: Skipped 1 previous similar message [ 7519.723916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7519.729174] Lustre: Skipped 1 previous similar message [ 7519.738172] Lustre: Skipped 1 previous similar message [ 7523.621833] Lustre: server umount lustre-OST0000 complete [ 7528.930328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7528.945090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.966039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7533.942669] Lustre: server umount lustre-MDT0000 complete [ 7535.073099] LustreError: 177609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.086390] LustreError: 177609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7537.911992] LustreError: 177595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775873689 with bad export cookie 17071489124639379592 [ 7537.915076] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7537.927982] LustreError: 177595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7538.293253] Lustre: server umount lustre-MDT0001 complete [ 7547.610835] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 22:14:56 (1775873696) [ 7565.878987] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 7568.541633] Key type lgssc unregistered [ 7568.796711] LNet: 182130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7568.804635] LNetError: 182130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7568.821943] LNet: Removed LNI 192.168.202.114@tcp [ 7569.570798] Key type .llcrypt unregistered [ 7569.573264] Key type ._llcrypt unregistered [ 7588.923675] Key type ._llcrypt registered [ 7588.925737] Key type .llcrypt registered [ 7589.043312] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7590.544110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7590.664187] alg: No test for adler32 (adler32-zlib) [ 7591.912037] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 7592.265321] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 7593.928950] Key type lgssc registered [ 7595.898333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7604.927904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7604.937114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.351372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.963742] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7616.481716] LustreError: 183330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7616.497159] LustreError: 183330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7619.725036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.047844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.184780] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7629.309858] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7632.960668] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7640.797211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.954153] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7640.956301] LustreError: 184970:0:(llog.c:1687:llog_backup()) MGC192.168.202.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7640.963829] Lustre: 184970:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.202.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7641.096667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.998182] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7652.285456] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7655.907664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7655.930471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.952743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7660.777493] Lustre: server umount lustre-MDT0000 complete [ 7661.543133] LustreError: 183325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.558233] LustreError: 183325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7664.135300] LustreError: 183311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775873815 with bad export cookie 9333141799597976041 [ 7664.155223] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7664.403328] Lustre: server umount lustre-MDT0001 complete [ 7678.733437] Lustre: server umount lustre-OST0000 complete [ 7688.178264] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7699.832187] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7736.958716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.252353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7737.293079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7737.379936] Lustre: lustre-MDT0000: new disk, initializing [ 7737.478385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7737.499537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7742.303881] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7752.498158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.619740] Lustre: 188357:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7752.667015] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7752.669186] Lustre: Skipped 1 previous similar message [ 7752.670900] LustreError: 189122:0:(llog.c:1687:llog_backup()) MGC192.168.202.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7752.679872] LustreError: 189122:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7752.685978] Lustre: 189122:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7752.708155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7752.716717] Lustre: Skipped 1 previous similar message [ 7752.809161] Lustre: lustre-MDT0001: new disk, initializing [ 7752.943990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7752.981617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7752.995157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7757.672555] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7764.620775] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7768.252080] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7774.685141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7775.028767] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7775.030460] Lustre: Skipped 1 previous similar message [ 7775.040097] LustreError: 190075:0:(llog.c:1687:llog_backup()) MGC192.168.202.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7775.053121] LustreError: 190075:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7775.063611] Lustre: 190075:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7775.079839] Lustre: 190075:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7775.129040] Lustre: lustre-OST0000: new disk, initializing [ 7775.134870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7775.236860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7780.767803] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7783.022805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7783.029924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7783.134108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7787.473797] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7791.621356] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7791.891946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7796.007311] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7796.237940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7798.759679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7798.763159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.767822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.786097] Lustre: Skipped 3 previous similar messages [ 7798.798812] Lustre: Skipped 4 previous similar messages [ 7803.876046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7803.890491] Lustre: Skipped 1 previous similar message [ 7809.010370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7809.015041] Lustre: Skipped 1 previous similar message [ 7812.063383] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7812.231848] Lustre: server umount lustre-OST0000 complete [ 7819.237641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.240625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.248291] Lustre: Skipped 1 previous similar message [ 7819.261587] Lustre: Skipped 1 previous similar message [ 7821.634601] Lustre: server umount lustre-MDT0000 complete [ 7824.356795] LustreError: 188369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.387536] LustreError: 188369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7824.901870] LustreError: 188349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775873976 with bad export cookie 9333141799597976762 [ 7824.909940] LustreError: 188349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7824.910396] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.314148] Lustre: server umount lustre-MDT0001 complete [ 7833.090351] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 22:19:42 (1775873982) [ 7848.079700] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7856.818787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.130436] LustreError: 193060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7857.191356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.543151] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7862.248633] LustreError: 193061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.161579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.657678] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7875.328528] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7881.052789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.546917] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7894.174564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7894.436566] Lustre: lustre-OST0001: new disk, initializing [ 7894.448930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7894.547377] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7894.551396] Lustre: Skipped 2 previous similar messages [ 7899.926943] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7901.763948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7901.776609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7901.818297] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7910.848881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7916.248811] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7923.187732] Lustre: DEBUG MARKER: ADD OST3 [ 7927.814220] loop: module loaded [ 7927.930774] loop0: detected capacity change from 0 to 204800000 [ 7928.138872] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7928.158054] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7928.176664] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7933.727942] loop0: detected capacity change from 0 to 204800000 [ 7934.666478] Lustre: DEBUG MARKER: STOP OST3 [ 7939.230324] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7939.486741] Lustre: lustre-OST0003: new disk, initializing [ 7939.497803] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7941.328420] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7941.439946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7941.453981] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7941.514649] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7942.898690] Lustre: Failing over lustre-OST0003 [ 7942.978936] Lustre: server umount lustre-OST0003 complete [ 7943.392449] LustreError: 197492:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7945.628919] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7946.208713] LustreError: 194510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7946.222505] LustreError: 194510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7946.733290] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7946.743011] LustreError: Skipped 1 previous similar message [ 7946.749936] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.472060] Lustre: DEBUG MARKER: START OST3 again [ 7959.392144] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7959.718730] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7959.724556] Lustre: Skipped 1 previous similar message [ 7959.738197] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7961.019516] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7961.517818] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7961.524965] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7965.339995] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7984.892389] Lustre: Failing over lustre-OST0003 [ 7984.957981] Lustre: server umount lustre-OST0003 complete [ 7985.645768] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7985.658586] Lustre: Skipped 1 previous similar message [ 7985.661957] LustreError: 195832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7985.668016] LustreError: 195832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 8006.115950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.118926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8006.130641] Lustre: Skipped 3 previous similar messages [ 8006.142430] Lustre: Skipped 3 previous similar messages [ 8016.358602] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8016.620637] Lustre: server umount lustre-MDT0000 complete [ 8019.660061] LustreError: 193041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775874171 with bad export cookie 9333141799597977644 [ 8019.662828] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8019.675192] LustreError: 193041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8021.472565] LustreError: 195766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8021.504252] LustreError: 195766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 8024.481578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8024.484042] Lustre: Skipped 10 previous similar messages [ 8025.993685] Lustre: server umount lustre-MDT0001 complete [ 8029.284192] Lustre: server umount lustre-OST0000 complete [ 8032.845918] Lustre: server umount lustre-OST0001 complete [ 8038.776458] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8044.935809] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8074.101431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8074.366204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8074.393101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8074.459833] Lustre: lustre-MDT0000: new disk, initializing [ 8074.565153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8078.352472] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8086.806250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.904250] Lustre: 202702:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8087.124473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8090.303501] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8096.469955] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8099.808848] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8105.588349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.801894] Lustre: lustre-OST0000: new disk, initializing [ 8105.804443] Lustre: Skipped 1 previous similar message [ 8105.808253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8105.817052] Lustre: Skipped 2 previous similar messages [ 8105.890678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.893403] Lustre: Skipped 2 previous similar messages [ 8107.587030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8107.593936] Lustre: Skipped 1 previous similar message [ 8107.598951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8107.719603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8111.000342] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8117.463913] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8120.739325] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8120.911289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8124.663404] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8124.866199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8128.480625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8128.486509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8128.498487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8128.505758] Lustre: Skipped 4 previous similar messages [ 8128.519906] Lustre: Skipped 2 previous similar messages [ 8132.892698] Lustre: server umount lustre-OST0000 complete [ 8138.210811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8138.215360] LustreError: Skipped 1 previous similar message [ 8142.509637] Lustre: server umount lustre-MDT0000 complete [ 8143.841328] LustreError: 203561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.857850] LustreError: 203561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8145.650413] LustreError: 202692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775874297 with bad export cookie 9333141799597979940 [ 8145.654512] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8145.663906] LustreError: 202692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8145.874519] Lustre: server umount lustre-MDT0001 complete [ 8149.320786] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 22:24:58 (1775874298) [ 8163.765668] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8171.657560] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8200.210572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8200.389507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8200.404817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8200.465882] Lustre: lustre-MDT0000: new disk, initializing [ 8200.539309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8203.797840] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8212.111164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.199431] Lustre: 208864:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8212.431511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8216.138350] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8222.544701] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8225.367935] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8230.572950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8232.054641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8235.470914] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8241.488762] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8244.876809] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8245.054811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8248.075096] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8248.289473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8252.388636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8252.391126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.397143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8252.397153] Lustre: Skipped 2 previous similar messages [ 8252.400425] LustreError: Skipped 1 previous similar message [ 8252.442482] Lustre: Skipped 3 previous similar messages [ 8264.159174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8264.371144] Lustre: server umount lustre-OST0000 complete [ 8273.827343] Lustre: server umount lustre-MDT0000 complete [ 8277.038830] LustreError: 210605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775874428 with bad export cookie 9333141799597980822 [ 8277.040239] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.048452] LustreError: 210605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8277.372278] Lustre: server umount lustre-MDT0001 complete [ 8284.496755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.903237] LustreError: 212675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8288.315453] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8295.842422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.962744] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8304.291389] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8307.070729] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8312.411550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8317.400494] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8322.192735] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8345.568369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8345.571341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.600684] Lustre: Skipped 3 previous similar messages [ 8347.935337] Lustre: server umount lustre-OST0000 complete [ 8357.801668] Lustre: server umount lustre-MDT0000 complete [ 8361.293469] LustreError: 212657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775874512 with bad export cookie 9333141799597981704 [ 8361.299836] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.308063] LustreError: 212657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8361.624805] Lustre: server umount lustre-MDT0001 complete [ 8462.062868] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 8464.651468] Key type lgssc unregistered [ 8464.946574] LNet: 216014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8464.953235] LNetError: 216014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8464.972230] LNet: Removed LNI 192.168.202.114@tcp [ 8465.663159] Key type .llcrypt unregistered [ 8465.665307] Key type ._llcrypt unregistered [ 8489.161585] Key type ._llcrypt registered [ 8489.164305] Key type .llcrypt registered [ 8489.237454] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8503.072373] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8504.171587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8504.237678] alg: No test for adler32 (adler32-zlib) [ 8505.276126] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 8505.466982] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 8507.104238] Key type lgssc registered [ 8508.101675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8537.167962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8537.188876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.393914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8538.421777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8538.489832] Lustre: lustre-MDT0000: new disk, initializing [ 8538.559840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8538.574687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8541.907458] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8551.079612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8551.191375] Lustre: 219336:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8551.307806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8551.312496] Lustre: Skipped 1 previous similar message [ 8551.345827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8551.394794] Lustre: lustre-MDT0001: new disk, initializing [ 8551.471623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8551.493642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8551.500221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8555.333966] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8561.982743] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8565.824399] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8572.604250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.847867] Lustre: lustre-OST0000: new disk, initializing [ 8572.852857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8572.965907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.529702] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8583.225104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8583.242723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8583.333571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8586.854500] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8590.965375] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8591.155660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8595.813449] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8596.046144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8599.014847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8599.017461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.022594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8599.032462] LustreError: Skipped 1 previous similar message [ 8599.045605] Lustre: Skipped 1 previous similar message [ 8603.803481] Lustre: server umount lustre-OST0000 complete [ 8607.712083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8607.716815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.726234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8607.733064] Lustre: Skipped 1 previous similar message [ 8609.249447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8613.440643] Lustre: server umount lustre-MDT0000 complete [ 8616.385907] LustreError: 219327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775874767 with bad export cookie 2397181719202794972 [ 8616.388783] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8616.393416] LustreError: 219327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8616.623614] Lustre: server umount lustre-MDT0001 complete [ 8631.303716] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8640.413640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.856544] LustreError: 223845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8640.870195] LustreError: 223845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8640.912718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8644.957919] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8646.115963] LustreError: 223846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.232456] LustreError: 223845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.322980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8652.613025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8656.170062] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8658.612112] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8663.905462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8664.252297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8669.951539] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8677.470785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8677.662375] Lustre: lustre-OST0001: new disk, initializing [ 8677.667057] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8683.546844] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8685.168034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8685.184470] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8685.314748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8694.477673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8704.688318] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8720.107366] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:34:29 (1775874869) [ 8724.458820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8724.472327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8724.487776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8724.491540] Lustre: Skipped 2 previous similar messages [ 8727.869553] Lustre: server umount lustre-MDT0000 complete [ 8730.684673] LustreError: 223827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775874882 with bad export cookie 2397181719202795854 [ 8730.690558] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8730.694512] LustreError: 223827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8731.036461] Lustre: server umount lustre-MDT0001 complete [ 8744.207015] Lustre: server umount lustre-OST0000 complete [ 8757.491344] Lustre: server umount lustre-OST0001 complete [ 8762.785447] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8770.383344] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8796.877129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.128380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8797.163395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8797.230441] Lustre: lustre-MDT0000: new disk, initializing [ 8797.313233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.316441] Lustre: Skipped 1 previous similar message [ 8797.337164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8800.197638] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8808.901795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8808.999806] Lustre: 230015:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8809.056849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8809.062084] Lustre: Skipped 1 previous similar message [ 8809.144360] Lustre: lustre-MDT0001: new disk, initializing [ 8809.261617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8809.276319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8812.392569] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8818.465406] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8821.318432] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8826.463474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8826.641662] Lustre: lustre-OST0000: new disk, initializing [ 8826.645360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8827.765721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8827.778337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8827.871282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8831.732091] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8837.595465] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8840.968209] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8841.133303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8844.587150] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8844.736996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8848.356893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.357703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8848.366796] Lustre: Skipped 3 previous similar messages [ 8848.372380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8848.386414] Lustre: Skipped 4 previous similar messages [ 8852.417881] Lustre: server umount lustre-OST0000 complete [ 8855.520967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8858.592606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8858.601934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8858.605847] Lustre: Skipped 1 previous similar message [ 8858.625232] Lustre: Skipped 2 previous similar messages [ 8861.397290] Lustre: server umount lustre-MDT0000 complete [ 8863.713086] LustreError: 230807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.732711] LustreError: 230807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8864.481733] LustreError: 230006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775875015 with bad export cookie 2397181719202797996 [ 8864.494158] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8864.498068] LustreError: 230006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8864.839607] Lustre: server umount lustre-MDT0001 complete [ 8871.306143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.623992] LustreError: 233824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.703225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.708520] Lustre: Skipped 2 previous similar messages [ 8875.238476] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8881.228653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.696837] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8889.087283] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8891.639172] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8896.399968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8900.693411] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8904.635567] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9051.616579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.621728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.630449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9056.794593] Lustre: server umount lustre-OST0000 complete [ 9065.547877] Lustre: server umount lustre-MDT0000 complete [ 9068.122544] LustreError: 233804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775875219 with bad export cookie 2397181719202798871 [ 9068.123438] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9068.135687] LustreError: 233804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9068.374701] Lustre: server umount lustre-MDT0001 complete [ 9169.058912] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 9170.852123] Key type lgssc unregistered [ 9171.062480] LNet: 237075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9171.067145] LNetError: 237075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9171.078534] LNet: Removed LNI 192.168.202.114@tcp [ 9171.688647] Key type .llcrypt unregistered [ 9171.691900] Key type ._llcrypt unregistered [ 9188.378607] Key type ._llcrypt registered [ 9188.380839] Key type .llcrypt registered [ 9188.443576] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9198.323462] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9199.079548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9199.101167] alg: No test for adler32 (adler32-zlib) [ 9200.063572] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [ 9200.246502] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 9201.887453] Key type lgssc registered [ 9202.679829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9231.211760] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9237.933350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9237.963920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.124420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9239.145805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9239.196495] Lustre: lustre-MDT0000: new disk, initializing [ 9239.317715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.337268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9242.210396] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9249.450636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.549515] Lustre: 241068:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9249.590187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9249.593819] Lustre: Skipped 1 previous similar message [ 9249.633662] Lustre: lustre-MDT0001: new disk, initializing [ 9249.682419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9249.714617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9249.722943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9252.093831] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9255.555702] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9259.821212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9260.011753] Lustre: lustre-OST0000: new disk, initializing [ 9260.019673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9260.040116] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9260.112690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9264.053166] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9265.167456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9265.183598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9265.207508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 9271.655656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9271.799602] Lustre: lustre-OST0001: new disk, initializing [ 9271.803085] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9271.868044] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9272.080212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9272.085606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9272.149873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9275.886179] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9284.784248] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9290.065656] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9303.015925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.021987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9303.024681] Lustre: Skipped 3 previous similar messages [ 9303.030351] Lustre: Skipped 3 previous similar messages [ 9307.745838] Lustre: server umount lustre-MDT0000 complete [ 9310.279061] LustreError: 244329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775875461 with bad export cookie 1565898745262878035 [ 9310.287374] LustreError: 244329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9310.287620] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9310.540799] Lustre: server umount lustre-MDT0001 complete [ 9323.589555] Lustre: server umount lustre-OST0000 complete [ 9336.630640] Lustre: server umount lustre-OST0001 complete [ 9346.465530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9352.868547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9358.100304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9364.440884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9376.176864] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9381.968640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9381.979791] Lustre: lustre-MDT0000: reset Object Index mappings [ 9382.260594] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.274778] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9382.337841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.784111] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9387.492456] LustreError: 248496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.985462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.001288] Lustre: lustre-MDT0001: reset Object Index mappings [ 9392.831524] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9394.649721] Lustre: 249602:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9398.309656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9398.514117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.518763] Lustre: Skipped 1 previous similar message [ 9400.548882] LustreError: 249956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.869297] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9405.929381] LustreError: 249958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.612059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9407.795593] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9407.802136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 9409.890527] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9414.597387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9416.646846] Lustre: 251444:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9426.409745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9426.416468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9426.429309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9428.451885] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.455939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9428.458530] Lustre: Skipped 1 previous similar message [ 9428.465107] Lustre: Skipped 3 previous similar messages [ 9430.437141] Lustre: server umount lustre-MDT0000 complete [ 9433.029990] LustreError: 249604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775875584 with bad export cookie 1565898745262884818 [ 9433.032799] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9433.038099] LustreError: 249604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9433.207839] Lustre: server umount lustre-MDT0001 complete [ 9445.982347] Lustre: server umount lustre-OST0000 complete [ 9458.482868] Lustre: server umount lustre-OST0001 complete [ 9470.256825] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:47:00 (1775875620) [ 9479.822851] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9485.200557] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9504.075177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.211384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9504.227996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9504.274360] Lustre: lustre-MDT0000: new disk, initializing [ 9504.328651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9504.333698] Lustre: Skipped 1 previous similar message [ 9504.344672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9506.580855] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9512.870551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9512.942286] Lustre: 255659:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9512.979494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9512.983508] Lustre: Skipped 1 previous similar message [ 9513.037483] Lustre: lustre-MDT0001: new disk, initializing [ 9513.120930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9513.129101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9515.501590] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9520.419427] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9522.653791] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9526.804341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9528.792943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9528.898420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9530.429344] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9534.942970] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9537.451812] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9537.580126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9539.811730] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9539.956327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9544.160260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9544.160479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.167067] LustreError: Skipped 1 previous similar message [ 9544.168805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9544.177062] Lustre: Skipped 2 previous similar messages [ 9547.146682] Lustre: server umount lustre-OST0000 complete [ 9549.280948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9549.283000] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9549.295736] Lustre: Skipped 1 previous similar message [ 9554.401455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9554.404195] Lustre: Skipped 4 previous similar messages [ 9555.371994] Lustre: server umount lustre-MDT0000 complete [ 9557.315680] LustreError: 255652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775875708 with bad export cookie 1565898745262886918 [ 9557.317694] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.327028] LustreError: 255652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9557.540853] Lustre: server umount lustre-MDT0001 complete [ 9567.695818] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9573.505138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.763470] LustreError: 260164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9573.795396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9573.798728] Lustre: Skipped 2 previous similar messages [ 9576.076860] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9581.188265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.931891] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9585.503270] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9589.091177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9593.547674] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9599.016381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9599.109209] Lustre: lustre-OST0001: new disk, initializing [ 9599.110858] Lustre: Skipped 1 previous similar message [ 9599.114330] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9599.117015] Lustre: Skipped 1 previous similar message [ 9601.229093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9601.232336] Lustre: Skipped 1 previous similar message [ 9601.235201] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9601.295138] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9602.589414] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9609.239941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9611.266707] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9619.637144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9619.822738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9627.107743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9627.112363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9627.120244] Lustre: Skipped 3 previous similar messages [ 9627.124124] Lustre: Skipped 2 previous similar messages [ 9630.063345] Lustre: server umount lustre-MDT0000 complete [ 9632.227701] LustreError: 260159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.246476] LustreError: 260159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9632.594799] LustreError: 260146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775875783 with bad export cookie 1565898745262887800 [ 9632.601428] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9632.604262] LustreError: 260146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9632.875591] Lustre: server umount lustre-MDT0001 complete [ 9645.591351] Lustre: server umount lustre-OST0000 complete [ 9657.720355] Lustre: server umount lustre-OST0001 complete [ 9667.846763] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9673.879561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.132878] LustreError: 265444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9674.186306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9674.191286] Lustre: Skipped 3 previous similar messages [ 9676.849462] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9682.957941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.458628] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9688.808518] Lustre: 266550:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9693.267716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9697.473434] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9698.691669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9703.574390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9705.290451] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9707.780413] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9712.879933] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9715.088918] Lustre: 268399:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9723.677614] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9723.803454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9730.082403] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:51:19 (1775875879) [ 9744.873101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9744.880414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9744.889138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9744.907118] Lustre: Skipped 3 previous similar messages [ 9748.577914] Lustre: server umount lustre-MDT0000 complete [ 9750.002555] LustreError: 265440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.011457] LustreError: 265440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9751.080286] LustreError: 265425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775875902 with bad export cookie 1565898745262894114 [ 9751.083637] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9751.089043] LustreError: 265425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9757.410740] Lustre: server umount lustre-MDT0001 complete [ 9766.392065] Lustre: server umount lustre-OST0000 complete [ 9774.639561] Lustre: server umount lustre-OST0001 complete [ 9777.835932] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9783.438938] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9812.577850] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9819.425487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9819.638747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9819.671071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9819.727453] Lustre: lustre-MDT0000: new disk, initializing [ 9819.799189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9819.800982] Lustre: Skipped 3 previous similar messages [ 9819.810740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9822.698478] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9831.226295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.291190] Lustre: 272582:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9831.459676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9834.899695] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9842.708038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9844.691539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9846.635366] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9853.673966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9855.584086] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9857.980369] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9865.137960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9867.437919] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9867.443994] Lustre: Skipped 1 previous similar message [ 9881.287877] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 22:53:51 (1775876031) [ 9886.176475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.184910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9886.191513] Lustre: Skipped 5 previous similar messages [ 9886.199833] Lustre: Skipped 10 previous similar messages [ 9888.869486] Lustre: server umount lustre-MDT0000 complete [ 9891.230647] LustreError: 272573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876042 with bad export cookie 1565898745262902003 [ 9891.235863] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9891.237940] LustreError: 272573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9891.297770] LustreError: 275175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.308879] LustreError: 275175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9891.451949] Lustre: server umount lustre-MDT0001 complete [ 9899.955582] Lustre: server umount lustre-OST0000 complete [ 9902.498883] Lustre: server umount lustre-OST0001 complete [ 9914.108344] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9920.239837] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9944.050151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.192892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9944.216846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9944.220732] Lustre: Skipped 4 previous similar messages [ 9944.287740] Lustre: lustre-MDT0000: new disk, initializing [ 9944.290273] Lustre: Skipped 3 previous similar messages [ 9944.348573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9944.353627] Lustre: Skipped 3 previous similar messages [ 9947.181094] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9954.311823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.362672] Lustre: 279615:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9954.490534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9954.497513] Lustre: Skipped 2 previous similar messages [ 9956.637094] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9961.090107] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9962.979622] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9966.592316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9968.242885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9969.874286] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9974.500248] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9975.684897] Lustre: Mounted lustre-client [ 9976.194258] LustreError: 282111:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8f66872e9000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9976.243709] Lustre: Unmounted lustre-client [ 9983.456349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9983.463097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9983.474509] Lustre: Skipped 2 previous similar messages [ 9987.377307] Lustre: server umount lustre-OST0000 complete [ 9995.954231] Lustre: server umount lustre-MDT0000 complete [ 9998.238977] LustreError: 280400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876149 with bad export cookie 1565898745262903529 [ 9998.240668] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9998.244548] LustreError: 280400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9998.443905] Lustre: server umount lustre-MDT0001 complete [10004.341769] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10005.877916] Key type lgssc unregistered [10034.719974] LNet: 283389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10034.733519] LNetError: 283389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10106.408334] LNet: Removed LNI 192.168.202.114@tcp [10106.867212] Key type .llcrypt unregistered [10106.869288] Key type ._llcrypt unregistered [10116.269181] Key type ._llcrypt registered [10116.271319] Key type .llcrypt registered [10116.330891] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10124.865086] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10125.337335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10125.393651] alg: No test for adler32 (adler32-zlib) [10126.293500] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [10126.400323] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10127.999205] Key type lgssc registered [10128.556409] Lustre: Echo OBD driver; http://www.lustre.org/ [10144.072292] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:58:14 (1775876294) [10147.205952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10147.216361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.319734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10148.334401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10148.371864] Lustre: lustre-MDT0000: new disk, initializing [10148.407403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10148.415421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10149.945133] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10155.031531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10155.094752] Lustre: 286550:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10155.139985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10155.142280] Lustre: Skipped 1 previous similar message [10155.193909] Lustre: lustre-MDT0001: new disk, initializing [10155.251731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10155.275512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10155.282208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10156.887965] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10160.435776] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10161.879886] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10164.472347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.584024] Lustre: lustre-OST0000: new disk, initializing [10164.586251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10164.628785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10164.679717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10164.683488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10164.699411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [10167.041215] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10170.826034] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10177.696584] Lustre: server umount lustre-OST0000 complete [10180.065637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10180.072750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10180.077728] Lustre: Skipped 1 previous similar message [10185.184714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10185.187194] Lustre: Skipped 1 previous similar message [10189.310881] Lustre: server umount lustre-MDT0000 complete [10190.852449] LustreError: 286543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876342 with bad export cookie 12813644972726599716 [10190.854863] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10190.856925] LustreError: 286543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10201.109733] Lustre: server umount lustre-MDT0001 complete [10205.119057] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10206.349857] Key type lgssc unregistered [10206.478628] LNet: 289923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10206.482387] LNetError: 289923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10206.492679] LNet: Removed LNI 192.168.202.114@tcp [10206.833230] Key type .llcrypt unregistered [10206.834465] Key type ._llcrypt unregistered [10217.791347] Key type ._llcrypt registered [10217.792478] Key type .llcrypt registered [10217.836902] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10218.179502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10218.186303] alg: No test for adler32 (adler32-zlib) [10219.050124] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [10219.153592] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10220.735172] Key type lgssc registered [10221.183334] Lustre: Echo OBD driver; http://www.lustre.org/ [10224.194359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10224.201665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10225.404186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10226.833976] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10230.215985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.383770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10231.868423] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10233.956761] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10235.414344] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10238.078853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10238.213728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10240.403494] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10242.398338] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10256.650488] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10257.248305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10258.876582] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.14@tcp (stopping) [10260.828540] Lustre: server umount lustre-MDT0000 complete [10260.961511] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.963285] LustreError: 291299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.963871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10260.967151] Lustre: Skipped 2 previous similar messages [10260.979410] LustreError: 291299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10262.288129] LustreError: 291281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876413 with bad export cookie 10038159235701793077 [10262.291187] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10262.292602] LustreError: 291281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10263.989367] LustreError: 291296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.997232] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [10264.474800] Lustre: server umount lustre-MDT0001 complete [10282.328652] Lustre: server umount lustre-OST0000 complete [10285.819250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.303305] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10288.422734] Lustre: 295047:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.114@tcp [10289.148938] Lustre: server umount MGS complete [10292.356469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.529626] LustreError: 295618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10292.554976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10294.007715] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10297.167959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10297.313957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10298.787440] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10301.836594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10304.022934] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10306.004795] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10306.592081] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10307.872848] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10315.650392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [10317.281497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10317.281702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.283540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10317.290362] Lustre: Skipped 1 previous similar message [10322.401525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10322.404108] Lustre: Skipped 2 previous similar messages [10322.968394] Lustre: server umount lustre-OST0000 complete [10327.520613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.521183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10327.524706] Lustre: Skipped 1 previous similar message [10330.392756] Lustre: server umount lustre-MDT0000 complete [10331.683153] LustreError: 297787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876483 with bad export cookie 10038159235701793903 [10331.686052] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10331.689226] LustreError: 297787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10331.813150] Lustre: server umount lustre-MDT0001 complete [10335.327016] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10336.368358] Key type lgssc unregistered [10336.486340] LNet: 298807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10336.488638] LNetError: 298807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10336.497320] LNet: Removed LNI 192.168.202.114@tcp [10336.782438] Key type .llcrypt unregistered [10336.783551] Key type ._llcrypt unregistered [10342.775808] Key type ._llcrypt registered [10342.777229] Key type .llcrypt registered [10342.812675] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10349.251073] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10349.576834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10349.597980] alg: No test for adler32 (adler32-zlib) [10350.453851] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [10350.536231] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10352.119151] Key type lgssc registered [10352.502844] Lustre: Echo OBD driver; http://www.lustre.org/ [10364.783794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10364.792884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.880532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10365.893208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10365.929712] Lustre: lustre-MDT0000: new disk, initializing [10365.956071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10365.963658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10367.338828] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10371.693933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.739147] Lustre: 301915:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10371.763969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10371.767119] Lustre: Skipped 1 previous similar message [10371.808233] Lustre: lustre-MDT0001: new disk, initializing [10371.843511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10371.855238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10371.858496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10373.254085] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10376.518172] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10377.751686] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10380.070713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10380.162404] Lustre: lustre-OST0000: new disk, initializing [10380.164419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10380.192158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10382.148888] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10385.307551] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10386.769451] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10387.450953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10387.455693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10387.496696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10387.883686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10389.414439] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10389.496665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10392.545718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10392.547496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.549537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10392.554565] Lustre: Skipped 1 previous similar message [10396.304410] Lustre: server umount lustre-OST0000 complete [10397.664418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10397.664589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10397.664948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10397.664952] Lustre: Skipped 1 previous similar message [10397.668024] Lustre: Skipped 1 previous similar message [10397.673533] LustreError: Skipped 1 previous similar message [10402.784093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10402.788043] Lustre: Skipped 2 previous similar messages [10403.609321] Lustre: server umount lustre-MDT0000 complete [10404.794118] LustreError: 301910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876556 with bad export cookie 634920154591411751 [10404.795989] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10404.797258] LustreError: 301910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10404.908708] Lustre: server umount lustre-MDT0001 complete [10407.755881] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 23:02:38 (1775876558) [10412.893368] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10415.931632] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10427.851939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.938480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10427.950193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10427.985277] Lustre: lustre-MDT0000: new disk, initializing [10428.010589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10428.017586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10429.382987] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10433.534079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.573271] Lustre: 308232:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10433.591239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10433.592908] Lustre: Skipped 1 previous similar message [10433.628491] Lustre: lustre-MDT0001: new disk, initializing [10433.658987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10433.670174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10433.673405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10434.985119] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10438.090091] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10439.281633] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10441.488327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10443.421175] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10444.468319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10444.471774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10444.503160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10446.493242] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10457.806828] Lustre: server umount lustre-OST0000 complete [10459.615667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10459.617993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10459.621910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10465.248179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10465.250872] Lustre: Skipped 3 previous similar messages [10469.364116] Lustre: server umount lustre-MDT0000 complete [10470.639041] LustreError: 308225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876622 with bad export cookie 634920154591412619 [10470.642495] LustreError: 308225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10470.642670] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10481.168700] Lustre: server umount lustre-MDT0001 complete [10484.959738] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10486.094897] Key type lgssc unregistered [10486.228566] LNet: 311991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10486.233523] LNetError: 311991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10486.246287] LNet: Removed LNI 192.168.202.114@tcp [10486.566509] Key type .llcrypt unregistered [10486.567403] Key type ._llcrypt unregistered [10519.318732] Key type ._llcrypt registered [10519.320246] Key type .llcrypt registered [10519.364345] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10519.470321] Key type .llcrypt unregistered [10519.471798] Key type ._llcrypt unregistered [10546.919613] Key type ._llcrypt registered [10546.920653] Key type .llcrypt registered [10546.954134] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10547.056201] Key type .llcrypt unregistered [10547.057213] Key type ._llcrypt unregistered [10557.510425] Key type ._llcrypt registered [10557.511870] Key type .llcrypt registered [10557.548537] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10557.644497] Key type .llcrypt unregistered [10557.646068] Key type ._llcrypt unregistered [10568.157119] Key type ._llcrypt registered [10568.158258] Key type .llcrypt registered [10568.195323] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10568.297502] Key type .llcrypt unregistered [10568.298607] Key type ._llcrypt unregistered [10594.438743] Key type ._llcrypt registered [10594.440243] Key type .llcrypt registered [10594.478547] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10594.582384] Key type .llcrypt unregistered [10594.583694] Key type ._llcrypt unregistered [10605.459175] Key type ._llcrypt registered [10605.460161] Key type .llcrypt registered [10605.489883] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10605.587303] Key type .llcrypt unregistered [10605.588455] Key type ._llcrypt unregistered [10616.270286] Key type ._llcrypt registered [10616.271647] Key type .llcrypt registered [10616.308588] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10616.413324] Key type .llcrypt unregistered [10616.414319] Key type ._llcrypt unregistered [10627.334599] Key type ._llcrypt registered [10627.335549] Key type .llcrypt registered [10627.374471] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10627.486518] Key type .llcrypt unregistered [10627.487711] Key type ._llcrypt unregistered [10637.982228] Key type ._llcrypt registered [10637.983574] Key type .llcrypt registered [10638.021058] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10638.114419] Key type .llcrypt unregistered [10638.115533] Key type ._llcrypt unregistered [10648.765172] Key type ._llcrypt registered [10648.766465] Key type .llcrypt registered [10648.801533] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10648.894338] Key type .llcrypt unregistered [10648.895307] Key type ._llcrypt unregistered [10669.144971] Key type ._llcrypt registered [10669.146309] Key type .llcrypt registered [10669.185127] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10669.292209] Key type .llcrypt unregistered [10669.293344] Key type ._llcrypt unregistered [10680.306927] Key type ._llcrypt registered [10680.308307] Key type .llcrypt registered [10680.347484] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10680.446312] Key type .llcrypt unregistered [10680.447298] Key type ._llcrypt unregistered [10691.496773] Key type ._llcrypt registered [10691.497809] Key type .llcrypt registered [10691.538342] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10691.642270] Key type .llcrypt unregistered [10691.643461] Key type ._llcrypt unregistered [10710.153305] Key type ._llcrypt registered [10710.154939] Key type .llcrypt registered [10710.194664] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10716.341765] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10716.663983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10716.720331] alg: No test for adler32 (adler32-zlib) [10717.588221] Lustre: Lustre: Build Version: 2.17.52_3_gc81aec7 [10717.673470] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10719.255160] Key type lgssc registered [10719.611066] Lustre: Echo OBD driver; http://www.lustre.org/ [10731.265951] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10731.271854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10732.361122] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10732.372851] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10732.407478] Lustre: tst250fs-MDT0000: new disk, initializing [10732.432225] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10732.438981] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10733.795694] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10738.233528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.282213] Lustre: 331610:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10738.309400] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10738.311101] Lustre: Skipped 1 previous similar message [10738.346750] Lustre: tst250fs-MDT0001: new disk, initializing [10738.381954] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10738.397452] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10738.400607] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10739.755556] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10742.131204] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10744.618857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.740313] Lustre: tst250fs-OST0000: new disk, initializing [10744.742875] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10744.775579] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10746.888631] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10749.435067] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10749.439033] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10749.468552] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10750.989513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10751.049095] Lustre: tst250fs-OST0001: new disk, initializing [10751.051647] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10751.086437] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10752.949125] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10755.487285] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10758.321106] Lustre: 331613:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.114@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [10766.820510] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10766.822329] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10766.824924] Lustre: Skipped 3 previous similar messages [10766.827530] Lustre: Skipped 2 previous similar messages [10771.874325] Lustre: server umount tst250fs-MDT0000 complete [10773.169963] LustreError: 331600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775876924 with bad export cookie 5926523353353257787 [10773.172572] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10773.174581] LustreError: 331600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10773.283625] Lustre: server umount tst250fs-MDT0001 complete [10784.260168] Lustre: server umount tst250fs-OST0000 complete [10794.710226] Lustre: server umount tst250fs-OST0001 complete [10800.547212] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10803.087394] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10812.836696] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:09:23 (1775876963) [10813.383811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10818.597830] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10821.475492] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10831.424724] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10639 sec ========= 23:09:41 (1775876981) [10831.984016] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:09:42 (1775876982) === [10833.081994] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:09:43 (1775876983) === [10846.045014] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10847.122394] Key type lgssc unregistered [10847.265411] LNet: 343055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10847.267829] LNetError: 343055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10847.275282] LNet: Removed LNI 192.168.202.114@tcp [10847.578120] Key type .llcrypt unregistered [10847.579094] Key type ._llcrypt unregistered