[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483746385 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002263] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007485] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011038] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013077] *** VALIDATE selinux *** [ 0.022009] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026495] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] *** VALIDATE tmpfs *** [ 0.031390] *** VALIDATE proc *** [ 0.032227] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038014] Speculative Store Bypass: Vulnerable [ 0.041589] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.043844] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044736] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045031] ... version: 2 [ 0.046019] ... bit width: 48 [ 0.047018] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049021] ... max period: 00007fffffffffff [ 0.050020] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052315] rcu: Hierarchical SRCU implementation. [ 0.054359] smp: Bringing up secondary CPUs ... [ 0.055646] x86: Booting SMP configuration: [ 0.056042] .... node #0, CPUs: #1 #2 #3 [ 0.060293] smp: Brought up 1 node, 4 CPUs [ 0.062019] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234179] node 0 deferred pages initialised in 168ms [ 0.238011] devtmpfs: initialized [ 0.239264] x86/mm: Memory block size: 128MB [ 0.241787] gcov: version magic: 0x41383552 [ 0.243559] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.247127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.250482] pinctrl core: initialized pinctrl subsystem [ 0.253318] [ 0.253922] ************************************************************* [ 0.256021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.259020] ** ** [ 0.261020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.263020] ** ** [ 0.265020] ** This means that this kernel is built to expose internal ** [ 0.268021] ** IOMMU data structures, which may compromise security on ** [ 0.271019] ** your system. ** [ 0.273018] ** ** [ 0.276021] ** If you see this message and you are not debugging the ** [ 0.278017] ** kernel, report this immediately to your vendor! ** [ 0.280021] ** ** [ 0.282022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285018] ************************************************************* [ 0.287834] NET: Registered protocol family 16 [ 0.290521] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.296085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.300041] cpuidle: using governor menu [ 0.301855] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.304928] PCI: Using configuration type 1 for base access [ 0.307164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.316135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322175] cryptd: max_cpu_qlen set to 1000 [ 0.324212] ACPI: Added _OSI(Module Device) [ 0.326028] ACPI: Added _OSI(Processor Device) [ 0.328022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.330024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.336053] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343602] ACPI: Interpreter enabled [ 0.345090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.347018] ACPI: Using IOAPIC for interrupt routing [ 0.349194] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.353439] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.363708] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.366044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.369026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.373120] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.379235] acpiphp: Slot [2] registered [ 0.381113] acpiphp: Slot [5] registered [ 0.382161] acpiphp: Slot [6] registered [ 0.384212] acpiphp: Slot [7] registered [ 0.385173] acpiphp: Slot [8] registered [ 0.387189] acpiphp: Slot [9] registered [ 0.389187] acpiphp: Slot [10] registered [ 0.391194] acpiphp: Slot [3] registered [ 0.393129] acpiphp: Slot [4] registered [ 0.395151] acpiphp: Slot [11] registered [ 0.397117] acpiphp: Slot [12] registered [ 0.399125] acpiphp: Slot [13] registered [ 0.401133] acpiphp: Slot [14] registered [ 0.405146] acpiphp: Slot [15] registered [ 0.406149] acpiphp: Slot [16] registered [ 0.408165] acpiphp: Slot [17] registered [ 0.410152] acpiphp: Slot [18] registered [ 0.411181] acpiphp: Slot [19] registered [ 0.413143] acpiphp: Slot [20] registered [ 0.415100] acpiphp: Slot [21] registered [ 0.416145] acpiphp: Slot [22] registered [ 0.418167] acpiphp: Slot [23] registered [ 0.420265] acpiphp: Slot [24] registered [ 0.421166] acpiphp: Slot [25] registered [ 0.422099] acpiphp: Slot [26] registered [ 0.424151] acpiphp: Slot [27] registered [ 0.426139] acpiphp: Slot [28] registered [ 0.427219] acpiphp: Slot [29] registered [ 0.429147] acpiphp: Slot [30] registered [ 0.431152] acpiphp: Slot [31] registered [ 0.433121] PCI host bridge to bus 0000:00 [ 0.435044] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.437032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.439031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.442031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.444030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.446036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.447187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.450964] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.453328] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.463021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.468046] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.471045] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.473024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.478033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.480511] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.482783] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.486037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.488810] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.495018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.511019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.516018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.522862] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.532023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.538019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.555024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.566536] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.573024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.580027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.593021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.605000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.611028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.618021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.633032] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.643584] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.651024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.662030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.689027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.703000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.707018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.715022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.731031] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.741784] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.752020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.763022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.783021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.794000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.798488] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.801524] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.804439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.807292] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.812090] iommu: Default domain type: Passthrough [ 0.814591] SCSI subsystem initialized [ 0.817242] ACPI: bus type USB registered [ 0.819166] usbcore: registered new interface driver usbfs [ 0.821143] usbcore: registered new interface driver hub [ 0.824131] usbcore: registered new device driver usb [ 0.827230] pps_core: LinuxPPS API ver. 1 registered [ 0.829018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.834091] PTP clock support registered [ 0.836178] EDAC MC: Ver: 3.0.0 [ 0.838232] PCI: Using ACPI for IRQ routing [ 0.839880] NetLabel: Initializing [ 0.840015] NetLabel: domain hash size = 128 [ 0.841025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.842112] NetLabel: unlabeled traffic allowed by default [ 0.845142] vgaarb: loaded [ 0.847335] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.849026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.855382] clocksource: Switched to clocksource kvm-clock [ 0.963607] VFS: Disk quotas dquot_6.6.0 [ 0.965206] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.967642] *** VALIDATE ramfs *** [ 0.969097] *** VALIDATE hugetlbfs *** [ 0.970862] pnp: PnP ACPI init [ 0.974297] pnp: PnP ACPI: found 6 devices [ 0.992528] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.995416] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.997772] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.000427] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.003093] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.005835] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.009127] NET: Registered protocol family 2 [ 1.012373] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.018240] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.022204] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.028570] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.032259] TCP: Hash tables configured (established 65536 bind 65536) [ 1.036077] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.039470] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.042940] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.046115] NET: Registered protocol family 1 [ 1.048933] RPC: Registered named UNIX socket transport module. [ 1.051551] RPC: Registered udp transport module. [ 1.053462] RPC: Registered tcp transport module. [ 1.054977] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.057565] NET: Registered protocol family 44 [ 1.059556] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.062027] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.064453] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.067159] PCI: CLS 0 bytes, default 64 [ 1.068954] Unpacking initramfs... [ 2.520213] debug: unmapping init [mem 0xffff921bbcc54000-0xffff921bbffbffff] [ 2.527348] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.529292] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.532899] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.034619] Initialise system trusted keyrings [ 3.038359] Key type blacklist registered [ 3.040330] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.052519] zbud: loaded [ 3.056180] *** VALIDATE nfs *** [ 3.058182] *** VALIDATE nfs4 *** [ 3.060038] pstore: using deflate compression [ 3.064321] Platform Keyring initialized [ 3.165545] NET: Registered protocol family 38 [ 3.168247] Key type asymmetric registered [ 3.170094] Asymmetric key parser 'x509' registered [ 3.172307] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.175564] io scheduler mq-deadline registered [ 3.177403] io scheduler kyber registered [ 3.179196] io scheduler bfq registered [ 3.181622] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.185429] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.189555] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.193103] ACPI: Power Button [PWRF] [ 3.198694] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.204224] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.214121] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.220568] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.232973] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.262089] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.291656] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.297158] Non-volatile memory driver v1.3 [ 3.299115] Linux agpgart interface v0.103 [ 3.332911] virtio_blk virtio1: [vda] 133936 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.335798] vda: detected capacity change from 0 to 68575232 [ 3.350794] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.353513] vdb: detected capacity change from 0 to 1073741824 [ 3.367578] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.370925] vdc: detected capacity change from 0 to 2621440000 [ 3.386358] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.389979] vdd: detected capacity change from 0 to 2621440000 [ 3.405179] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.407652] vde: detected capacity change from 0 to 4294967296 [ 3.422135] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.424658] vdf: detected capacity change from 0 to 4294967296 [ 3.431577] libphy: Fixed MDIO Bus: probed [ 3.450376] usbcore: registered new interface driver usbserial_generic [ 3.452715] usbserial: USB Serial support registered for generic [ 3.455029] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.458745] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.460540] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.463805] mousedev: PS/2 mouse device common for all mice [ 3.467427] rtc_cmos 00:05: RTC can wake from S4 [ 3.470238] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.473044] rtc_cmos 00:05: registered as rtc0 [ 3.476290] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.481847] intel_pstate: CPU model not supported [ 3.482629] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.488591] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.488638] hid: raw HID events driver (C) Jiri Kosina [ 3.493849] usbcore: registered new interface driver usbhid [ 3.495332] usbhid: USB HID core driver [ 3.496478] drop_monitor: Initializing network drop monitor service [ 3.498727] Initializing XFRM netlink socket [ 3.500715] NET: Registered protocol family 10 [ 3.503097] Segment Routing with IPv6 [ 3.504468] NET: Registered protocol family 17 [ 3.506023] mpls_gso: MPLS GSO support [ 3.510198] RAS: Correctable Errors collector initialized. [ 3.511589] AVX version of gcm_enc/dec engaged. [ 3.513098] AES CTR mode by8 optimization enabled [ 3.581485] sched_clock: Marking stable (3581354095, 0)->(4530655184, -949301089) [ 3.585691] registered taskstats version 1 [ 3.588083] Loading compiled-in X.509 certificates [ 3.590367] zswap: loaded using pool lzo/zbud [ 3.615672] Key type big_key registered [ 3.627716] Key type encrypted registered [ 3.630116] ima: No TPM chip found, activating TPM-bypass! [ 3.632393] ima: Allocated hash algorithm: sha1 [ 3.634032] ima: No architecture policies found [ 3.637337] evm: Initialising EVM extended attributes: [ 3.639970] evm: security.selinux [ 3.642323] evm: security.ima [ 3.643309] evm: security.capability [ 3.644324] evm: HMAC attrs: 0x1 [ 3.648109] rtc_cmos 00:05: setting system clock to 2026-04-03 11:30:14 UTC (1775215814) [ 3.653906] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 3.656586] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 3.665190] Write protecting the kernel read-only data: 28672k [ 3.668739] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.671778] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 3.709548] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.718018] systemd[1]: Detected virtualization kvm. [ 3.719601] systemd[1]: Detected architecture x86-64. [ 3.721393] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.746415] systemd[1]: No hostname configured. [ 3.748237] systemd[1]: Set hostname to . [ 3.749685] random: systemd: uninitialized urandom read (16 bytes read) [ 3.752286] systemd[1]: Initializing machine ID from random generator. [ 3.781333] random: ln: uninitialized urandom read (6 bytes read) [ 3.863452] random: systemd: uninitialized urandom read (16 bytes read) [ 3.866414] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.870691] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.874488] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.499976] device-mapper: uevent: version 1.0.3 [ 4.502443] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 5.931736] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.996647] random: fast init done [ 6.196762] scsi host0: ata_piix [ 6.225369] scsi host1: ata_piix [ 6.230381] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.243093] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.153993] random: crng init done [ 11.158893] random: 7 urandom warning(s) missed due to ratelimiting [ 12.190281] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.380381] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.511316] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.513172] SELinux: Disabled at runtime. [ 18.678193] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.699600] systemd[1]: Detected virtualization kvm. [ 18.702204] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.082964] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.096419] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.116150] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.124859] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.133651] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.144363] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.168964] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ 20.894310] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.740300] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.688174] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.693477] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.808587] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.912256] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 28.762026] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.644512] NFS: Registering the id_resolver key type [ 29.651739] Key type id_resolver registered [ 29.653914] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg653-server login: [ 75.305056] hrtimer: interrupt took 4992270 ns [ 94.214516] libcfs: loading out-of-tree module taints kernel. [ 94.252985] Key type ._llcrypt registered [ 94.258819] Key type .llcrypt registered [ 94.358330] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 116.175182] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 117.704279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.711992] alg: No test for adler32 (adler32-zlib) [ 119.531759] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 120.880119] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 122.684337] Key type lgssc registered [ 125.128987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.786681] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.482182] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 198.131657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.160488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.412985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.439215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.567330] Lustre: lustre-MDT0000: new disk, initializing [ 199.660454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.698145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.494721] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.650771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.759605] Lustre: 6514:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.793330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.798545] Lustre: Skipped 1 previous similar message [ 219.942955] Lustre: lustre-MDT0001: new disk, initializing [ 220.020347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.045715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.062439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.000140] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.149817] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.796283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.117786] Lustre: lustre-OST0000: new disk, initializing [ 241.121642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.173744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.963531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.984800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.055993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.762364] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.735260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.925232] Lustre: lustre-OST0001: new disk, initializing [ 262.933198] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.014492] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.860042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.870070] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.948219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.084610] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.154926] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 286.598707] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 297.565848] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.943520] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing yml_node [ 306.128315] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 308.474884] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 310.928659] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 312.295551] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 3 07:35:21 EDT 2026 [ 330.435738] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 331.945639] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 348.128781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 348.137838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.155536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.690852] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.690971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.710192] Lustre: Skipped 2 previous similar messages [ 350.723797] Lustre: Skipped 3 previous similar messages [ 354.386996] Lustre: server umount lustre-MDT0000 complete [ 358.627613] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216169 with bad export cookie 11629911539755109378 [ 358.631197] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.639178] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 360.935163] LustreError: 9480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.939138] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.942378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.974235] LustreError: 9480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 360.992413] Lustre: Skipped 1 previous similar message [ 363.936271] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.936327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.971509] Lustre: Skipped 2 previous similar messages [ 365.303761] Lustre: server umount lustre-MDT0001 complete [ 369.257159] Lustre: server umount lustre-OST0000 complete [ 373.062782] Lustre: server umount lustre-OST0001 complete [ 379.765662] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 388.984618] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 422.868582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.150233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 423.220350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 423.326084] Lustre: lustre-MDT0000: new disk, initializing [ 423.458878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.514049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.743911] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 438.142662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.280172] Lustre: 14987:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 438.331629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 438.335366] Lustre: Skipped 1 previous similar message [ 438.389816] Lustre: lustre-MDT0001: new disk, initializing [ 438.487509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.529868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 438.539801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.945737] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 450.536645] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 454.452437] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 460.930844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 461.242055] Lustre: lustre-OST0000: new disk, initializing [ 461.251980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 461.342942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.793062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.819811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.953284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 467.372876] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 474.771595] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 479.259886] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 479.489210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 484.375302] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 484.572862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.417606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.422618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.422780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.441521] Lustre: Skipped 1 previous similar message [ 492.699702] Lustre: server umount lustre-OST0000 complete [ 498.658033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.662274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.685469] Lustre: Skipped 1 previous similar message [ 498.706895] Lustre: Skipped 2 previous similar messages [ 502.876950] Lustre: server umount lustre-MDT0000 complete [ 503.782414] LustreError: 16692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.802729] LustreError: 16692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 506.772220] LustreError: 14979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216317 with bad export cookie 11629911539755110911 [ 506.787262] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.791358] LustreError: 14979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 507.122192] Lustre: server umount lustre-MDT0001 complete [ 513.532443] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:38:42 (1775216322) [ 531.802893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.317976] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.402656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.368407] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 537.582319] LustreError: 19458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.198248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.719359] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 554.125445] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 558.222701] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 560.475686] Lustre: Failing over lustre-MDT0000 [ 560.717265] Lustre: server umount lustre-MDT0000 complete [ 563.168531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.177696] LustreError: 20469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.181256] Lustre: Skipped 1 previous similar message [ 563.217923] LustreError: 20469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 579.553247] Lustre: 3662:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775216374/real 1775216374] req@ffff921c0457c380 x1861448820220416/t0(0) o400->MGC192.168.206.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775216390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 579.557108] LustreError: 20615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.588367] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.623905] LustreError: 20615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 580.271743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.809559] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa165beed7051d785 [ 589.818490] Lustre: MGC192.168.206.153@tcp: Connection restored to 0@lo (at 0@lo) [ 593.877849] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 595.425650] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 595.440035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 595.490728] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 601.578833] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 605.667417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 605.676776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.690209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.331343] Lustre: server umount lustre-MDT0000 complete [ 614.817128] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216425 with bad export cookie 11629911539755112325 [ 614.821356] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.842116] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 615.308793] Lustre: server umount lustre-MDT0001 complete [ 623.428627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.949934] LustreError: 22455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.968507] LustreError: 22455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 624.024192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.028828] Lustre: Skipped 2 previous similar messages [ 628.201865] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 637.417981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.115330] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 647.365029] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 651.815609] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 654.252503] Lustre: Failing over lustre-MDT0000 [ 654.523345] Lustre: server umount lustre-MDT0000 complete [ 658.408730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.412124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 658.428745] Lustre: Skipped 2 previous similar messages [ 673.910985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.117898] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.420819] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 679.048456] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 679.403366] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 679.404954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 679.431767] Lustre: Skipped 1 previous similar message [ 679.449597] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 686.657869] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 694.758253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.768054] Lustre: Skipped 3 previous similar messages [ 697.220502] Lustre: server umount lustre-MDT0000 complete [ 699.873881] LustreError: 22455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.906100] LustreError: 22455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 701.184763] LustreError: 22435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216512 with bad export cookie 11629911539755113270 [ 701.195617] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.199777] LustreError: 22435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 701.553500] Lustre: server umount lustre-MDT0001 complete [ 709.607463] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:41:58 (1775216518) [ 727.360321] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 737.893280] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 776.788984] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 787.774506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.134265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 788.180468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 788.347400] Lustre: lustre-MDT0000: new disk, initializing [ 788.588754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.592266] Lustre: Skipped 2 previous similar messages [ 788.612807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 794.008402] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 805.175115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.342291] Lustre: 28453:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 805.731406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 810.869517] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 819.150429] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 823.356973] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 830.090032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 832.385347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 832.456936] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 832.519768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 832.554557] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 837.120592] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 845.062561] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 853.097881] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 872.509996] Lustre: server umount lustre-OST0000 complete [ 877.536794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 877.544400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.555324] Lustre: Skipped 2 previous similar messages [ 877.560136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 886.095213] Lustre: server umount lustre-MDT0000 complete [ 888.803785] LustreError: 29553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.817858] LustreError: 29553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 889.700104] Lustre: 3664:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775216684/real 1775216684] req@ffff921b01fc0e00 x1861448820309504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1775216700 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 890.569290] LustreError: 30193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216701 with bad export cookie 11629911539755113676 [ 890.574364] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.588274] LustreError: 30193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 890.853519] Lustre: server umount lustre-MDT0001 complete [ 903.904966] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 907.084691] Key type lgssc unregistered [ 907.337167] LNet: 32041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.345334] LNetError: 32041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.391668] LNet: Removed LNI 192.168.206.153@tcp [ 909.256134] Key type .llcrypt unregistered [ 909.257647] Key type ._llcrypt unregistered [ 933.607666] Key type ._llcrypt registered [ 933.611346] Key type .llcrypt registered [ 933.743361] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 934.838093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.019459] alg: No test for adler32 (adler32-zlib) [ 936.111922] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 936.370352] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 938.055167] Key type lgssc registered [ 939.234614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 947.810973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 947.826974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.302310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.719702] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 959.464816] LustreError: 33438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.480856] LustreError: 33438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 962.896696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.320853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.957905] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 973.875256] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 977.857963] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 983.756566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.272880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.407295] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 990.023926] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 994.709589] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1008.424246] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:46:57 (1775216817) [ 1010.367345] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1012.171817] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1021.504762] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:47:09 (1775216829) [ 1031.255466] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:47:19 (1775216839) [ 1032.371211] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1041.392763] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1041.398282] Lustre: Skipped 2 previous similar messages [ 1043.308136] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:47:31 (1775216851) [ 1047.203854] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1047.209575] Lustre: Skipped 3 previous similar messages [ 1064.326282] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1064.338472] Lustre: Skipped 6 previous similar messages [ 1067.068435] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:47:55 (1775216875) [ 1069.823628] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1073.129784] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1081.370583] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:48:10 (1775216890) [ 1083.431896] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1083.436944] Lustre: Skipped 3 previous similar messages [ 1096.994127] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:48:25 (1775216905) [ 1116.640977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.652674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.669697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.666186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.670529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1119.865077] Lustre: server umount lustre-OST0000 complete [ 1127.392570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1127.402309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.417303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.423366] Lustre: Skipped 1 previous similar message [ 1133.023946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.038804] Lustre: Skipped 2 previous similar messages [ 1138.143364] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1138.152627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.164359] Lustre: Skipped 1 previous similar message [ 1138.382220] Lustre: server umount lustre-MDT0000 complete [ 1142.215190] LustreError: 36663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216953 with bad export cookie 18263571404906891667 [ 1142.219687] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.230027] LustreError: 36663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1142.692484] Lustre: server umount lustre-MDT0001 complete [ 1154.421140] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 1157.396664] Key type lgssc unregistered [ 1157.799353] LNet: 40930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.815402] LNetError: 40930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.859319] LNet: Removed LNI 192.168.206.153@tcp [ 1158.810976] Key type .llcrypt unregistered [ 1158.820053] Key type ._llcrypt unregistered [ 1178.238669] Key type ._llcrypt registered [ 1178.243629] Key type .llcrypt registered [ 1178.365340] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 1194.456759] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 1195.773368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1195.849179] alg: No test for adler32 (adler32-zlib) [ 1196.908804] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1197.106449] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 1198.799541] Key type lgssc registered [ 1200.161301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.234869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1235.258250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.499042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1236.526410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1236.605690] Lustre: lustre-MDT0000: new disk, initializing [ 1236.669390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.691471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1241.802638] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1251.791110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.944183] Lustre: 44049:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1252.087289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1252.091897] Lustre: Skipped 1 previous similar message [ 1252.191358] Lustre: lustre-MDT0001: new disk, initializing [ 1252.316900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1252.362358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1252.375854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1256.046380] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1263.719572] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1267.938224] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1273.588890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.816172] Lustre: lustre-OST0000: new disk, initializing [ 1273.819949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1273.871433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.645441] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1283.189526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1283.202838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1283.401234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1286.391945] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1292.668754] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1292.823389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1297.813907] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1298.178502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1304.047813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1304.049304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.068756] LustreError: Skipped 1 previous similar message [ 1304.070955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1304.132375] Lustre: Skipped 1 previous similar message [ 1306.495532] Lustre: server umount lustre-OST0000 complete [ 1313.762382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.766487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.779885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.783789] Lustre: Skipped 1 previous similar message [ 1316.290608] Lustre: server umount lustre-MDT0000 complete [ 1320.467051] LustreError: 44041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217131 with bad export cookie 13057849611993085461 [ 1320.469776] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.472793] LustreError: 44041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1320.725861] Lustre: server umount lustre-MDT0001 complete [ 1329.071299] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:52:18 (1775217138) [ 1335.920878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.369411] LustreError: 48102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.389035] LustreError: 48102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1336.429281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.115230] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1341.922388] LustreError: 48103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.051809] LustreError: 48102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.108771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.505557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1355.383436] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1361.399920] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1365.990785] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1372.604561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.028645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.633839] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1385.120808] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1389.443544] LustreError: 50297:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1390.305338] Lustre: Modifying parameter general.timeout=1 in log params [ 1392.184615] Lustre: Modifying parameter general.timeout=3 in log params [ 1392.188526] Lustre: Skipped 1 previous similar message [ 1394.367737] Lustre: Modifying parameter general.timeout=6 in log params [ 1394.377718] Lustre: Skipped 2 previous similar messages [ 1398.612721] Lustre: Modifying parameter general.timeout=11 in log params [ 1398.617656] Lustre: Skipped 4 previous similar messages [ 1407.173811] Lustre: Modifying parameter general.timeout=21 in log params [ 1407.178654] Lustre: Skipped 9 previous similar messages [ 1423.353576] Lustre: Modifying parameter general.timeout=40 in log params [ 1423.358467] Lustre: Skipped 18 previous similar messages [ 1456.099052] Lustre: Modifying parameter general.timeout=77 in log params [ 1456.106165] Lustre: Skipped 36 previous similar messages [ 1515.896813] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:55:24 (1775217324) [ 1518.631633] Lustre: 57238:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1520.525933] Lustre: 57349:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1521.454424] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1521.463735] Lustre: Skipped 56 previous similar messages [ 1524.410034] Lustre: 57545:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1524.423543] Lustre: 57545:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1555.710116] Lustre: 59121:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1555.724372] Lustre: 59121:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1555.771318] LustreError: 59121:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1560.009645] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:56:08 (1775217368) [ 1575.905408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1575.919600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.942764] Lustre: Skipped 1 previous similar message [ 1575.953877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.966249] Lustre: Skipped 2 previous similar messages [ 1579.530635] Lustre: server umount lustre-MDT0000 complete [ 1582.053036] LustreError: 48098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.073822] LustreError: 48098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1583.276395] LustreError: 48083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217394 with bad export cookie 13057849611993086336 [ 1583.286545] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.296051] LustreError: 48083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1583.666406] Lustre: server umount lustre-MDT0001 complete [ 1597.403884] Lustre: server umount lustre-OST0000 complete [ 1624.058431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.504940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1625.032593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.926475] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1641.266872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.419623] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1641.474387] Lustre: 61652:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1646.908152] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1653.753358] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.777280] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1665.781967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1666.013143] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1666.377296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.387900] Lustre: Skipped 1 previous similar message [ 1673.384763] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1679.497942] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1681.227407] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1681.240280] Lustre: Skipped 19 previous similar messages [ 1681.294679] Lustre: 63880:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1691.105666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1691.117182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.135254] Lustre: Skipped 2 previous similar messages [ 1691.140517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.147167] Lustre: Skipped 3 previous similar messages [ 1695.009517] Lustre: server umount lustre-OST0000 complete [ 1701.857817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.871145] Lustre: Skipped 1 previous similar message [ 1701.876489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.882738] Lustre: Skipped 2 previous similar messages [ 1712.097316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.107676] Lustre: Skipped 3 previous similar messages [ 1713.120809] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1713.363746] Lustre: server umount lustre-MDT0000 complete [ 1717.226973] LustreError: 61662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.268047] LustreError: 61662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1718.343609] LustreError: 64051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217529 with bad export cookie 13057849611993089437 [ 1718.348906] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1718.363504] LustreError: 64051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1719.168600] Lustre: server umount lustre-MDT0001 complete [ 1731.268170] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 1734.590558] Key type lgssc unregistered [ 1735.135911] LNet: 65076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.168754] LNetError: 65076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.197494] LNet: Removed LNI 192.168.206.153@tcp [ 1736.438123] Key type .llcrypt unregistered [ 1736.440838] Key type ._llcrypt unregistered [ 1762.610008] Key type ._llcrypt registered [ 1762.615498] Key type .llcrypt registered [ 1762.734683] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 1764.311633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.354809] alg: No test for adler32 (adler32-zlib) [ 1765.512061] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1765.901715] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 1767.623766] Key type lgssc registered [ 1768.942456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1777.696098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1777.714911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.329573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.433534] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1789.929717] LustreError: 66474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1789.943729] LustreError: 66474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1790.950327] LustreError: 66473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.549671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.227949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.085894] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1806.725163] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.976645] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1818.116038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.819934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.772903] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1832.316811] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1841.623306] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1851.877107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.882659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.908748] Lustre: Skipped 1 previous similar message [ 1851.926285] Lustre: Skipped 1 previous similar message [ 1856.480654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1856.486613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.505125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1857.957792] Lustre: server umount lustre-MDT0000 complete [ 1861.862427] LustreError: 66455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217672 with bad export cookie 6403486874466182062 [ 1861.874279] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1861.885790] LustreError: 66455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1862.117151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.133390] LustreError: 66469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.442389] Lustre: server umount lustre-MDT0001 complete [ 1869.267650] Lustre: server umount lustre-OST0000 complete [ 1897.215763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.595765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1897.645596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1898.035461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.822890] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1912.235923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.410993] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1912.456629] Lustre: 71026:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1913.082592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.521800] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1924.795520] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1928.830530] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1936.826737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1937.033476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1937.378113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.145985] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 1951.417485] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1953.518961] Lustre: 73260:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1955.520665] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1965.026052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.038045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1965.041675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1965.041685] Lustre: Skipped 2 previous similar messages [ 1965.052405] Lustre: Skipped 1 previous similar message [ 1965.097037] LustreError: Skipped 1 previous similar message [ 1966.651616] Lustre: server umount lustre-OST0000 complete [ 1974.241055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1974.249352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.263581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.275441] Lustre: Skipped 1 previous similar message [ 1976.559610] Lustre: server umount lustre-MDT0000 complete [ 1980.387074] LustreError: 72038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.414132] LustreError: 72038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1980.744991] LustreError: 71017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217791 with bad export cookie 6403486874466182909 [ 1980.753779] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.761134] LustreError: 71017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1981.251565] Lustre: server umount lustre-MDT0001 complete [ 1995.642189] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 1999.212798] Key type lgssc unregistered [ 1999.634245] LNet: 74455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.645483] LNetError: 74455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.658509] LNet: Removed LNI 192.168.206.153@tcp [ 2000.650406] Key type .llcrypt unregistered [ 2000.654059] Key type ._llcrypt unregistered [ 2028.739513] Key type ._llcrypt registered [ 2028.742376] Key type .llcrypt registered [ 2028.867744] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 2029.788590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.868996] alg: No test for adler32 (adler32-zlib) [ 2031.439915] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2031.764500] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 2033.472205] Key type lgssc registered [ 2035.177795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.049204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2046.084654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.637867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.388649] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2057.187709] LustreError: 75853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.209931] LustreError: 75853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2059.886392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.529848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.076650] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2071.049271] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2075.412141] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2083.617656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2084.468789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.656993] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2097.376316] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2104.648855] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2105.650853] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2106.663681] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2109.425283] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2109.437459] Lustre: Skipped 2 previous similar messages [ 2113.955710] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2113.961428] Lustre: Skipped 4 previous similar messages [ 2122.075441] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2122.080032] Lustre: Skipped 8 previous similar messages [ 2138.403770] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2138.408734] Lustre: Skipped 15 previous similar messages [ 2171.165865] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2171.172863] Lustre: Skipped 32 previous similar messages [ 2210.665559] Lustre: 83625:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2213.384820] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:07:02 (1775218022) [ 2215.110698] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2217.164960] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:07:05 (1775218025) [ 2224.097094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2224.104750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.122807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.146853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2226.149170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.154968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.185530] Lustre: Skipped 1 previous similar message [ 2227.888188] Lustre: server umount lustre-OST0000 complete [ 2234.336426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2234.353371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.375739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.384914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.398901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.419448] Lustre: server umount lustre-MDT0000 complete [ 2241.504694] LustreError: 78565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.532577] LustreError: 78565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2242.185516] LustreError: 75833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218053 with bad export cookie 9786017296496360182 [ 2242.196751] LustreError: 75833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2242.196915] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.492970] Lustre: server umount lustre-MDT0001 complete [ 2253.261293] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 2256.430579] Key type lgssc unregistered [ 2256.745231] LNet: 84861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.750534] LNetError: 84861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.763326] LNet: Removed LNI 192.168.206.153@tcp [ 2257.399533] Key type .llcrypt unregistered [ 2257.402870] Key type ._llcrypt unregistered [ 2261.814876] Key type ._llcrypt registered [ 2261.822382] Key type .llcrypt registered [ 2261.989246] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2267.073569] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules [ 2268.485612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2268.623313] alg: No test for adler32 (adler32-zlib) [ 2269.964631] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2270.181943] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 2271.839554] Key type lgssc registered [ 2273.641806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2274.857560] LustreError: 86046:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 2291.051121] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 2295.908424] LustreError: 86046:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2301.552789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2301.582791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.233236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.969329] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2312.675161] LustreError: 87038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.697132] LustreError: 87038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2316.639781] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:08:45 (1775218125) [ 2317.798662] LustreError: 87037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.373381] LustreError: 87069:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2321.389468] LustreError: 87069:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2321.398537] LustreError: 87069:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2321.743540] Lustre: server umount lustre-MDT0000 complete [ 2334.181673] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 2337.060831] Key type lgssc unregistered [ 2337.395777] LNet: 88530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.407981] LNetError: 88530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.426080] LNet: Removed LNI 192.168.206.153@tcp [ 2338.261669] Key type .llcrypt unregistered [ 2338.265670] Key type ._llcrypt unregistered [ 2355.380664] Key type ._llcrypt registered [ 2355.383321] Key type .llcrypt registered [ 2355.476134] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 2356.988636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.045712] alg: No test for adler32 (adler32-zlib) [ 2358.231907] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2358.484218] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 2360.191191] Key type lgssc registered [ 2361.436281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.157351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2369.197450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.697134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.549070] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2380.260507] LustreError: 89731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.300214] LustreError: 89731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2385.158209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.378094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2386.092371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.797965] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2397.595959] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2402.615249] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2410.775901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2410.981548] LustreError: 91332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.895327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.230996] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2426.292340] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2479.011025] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:11:27 (1775218287) [ 2481.039319] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2483.187097] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:11:31 (1775218291) [ 2487.778085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.786471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.796378] Lustre: Skipped 2 previous similar messages [ 2492.896503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.914960] Lustre: Skipped 4 previous similar messages [ 2494.159471] Lustre: server umount lustre-MDT0000 complete [ 2498.018203] LustreError: 89726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.035489] LustreError: 89726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2498.129141] LustreError: 89712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218308 with bad export cookie 18331327033053022511 [ 2498.139523] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.467257] Lustre: server umount lustre-MDT0001 complete [ 2511.599470] Lustre: server umount lustre-OST0000 complete [ 2524.344281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.789043] LustreError: 93691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.023862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.275310] LustreError: 93692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.289014] LustreError: 93692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2530.308376] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2539.492286] LustreError: 93691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.520480] LustreError: 93691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2539.818046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.979810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2546.060444] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2551.953254] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2556.372093] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2571.773533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2572.019198] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2572.028170] LustreError: 93680:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2572.068719] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2572.089638] LustreError: 95549:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 2572.103650] LustreError: 95549:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2572.115212] LustreError: 95549:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2572.203905] Lustre: server umount lustre-OST0000 complete [ 2572.208520] LustreError: 95549:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2581.371453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2581.552303] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2581.572526] LustreError: 93680:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2581.598390] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2581.611274] LustreError: 96016:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 2581.620860] LustreError: 96016:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2581.628334] LustreError: 96016:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2581.687917] Lustre: server umount lustre-OST0000 complete [ 2581.691684] LustreError: 96016:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2591.431212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2591.629385] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2591.633673] Lustre: Found index 0 for lustre-OST0000, updating log [ 2591.637301] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2591.735964] Lustre: lustre-OST0000: new disk, initializing [ 2591.755595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2592.053773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.726234] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2601.678393] Lustre: Failing over lustre-OST0000 [ 2601.980352] Lustre: server umount lustre-OST0000 complete [ 2607.088599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2607.099629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.126037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2610.211996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.264085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.272300] Lustre: Skipped 3 previous similar messages [ 2616.393505] Lustre: server umount lustre-MDT0000 complete [ 2620.384247] LustreError: 93686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.397517] LustreError: 93686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2620.645161] LustreError: 93672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218431 with bad export cookie 18331327033053025248 [ 2620.664880] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2620.676366] LustreError: 93672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2621.055822] Lustre: server umount lustre-MDT0001 complete [ 2631.486836] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:13:59 (1775218439) [ 2650.280346] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 2662.216101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.748742] LustreError: 98870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.946580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.306845] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2677.658655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.101381] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2684.948085] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2691.113574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.935969] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2699.550959] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2702.183555] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2702.208313] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2702.229689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2707.317894] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2707.358047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2707.382089] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2707.382393] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2707.391570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2709.735875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.899976] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2710.056456] Lustre: lustre-OST0001: new disk, initializing [ 2710.073250] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2710.369423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2710.383878] Lustre: Skipped 2 previous similar messages [ 2716.849883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2716.864101] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2717.007595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2717.287987] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2724.938453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2736.640539] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2747.876150] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.902635] Lustre: Skipped 2 previous similar messages [ 2747.904966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2747.916583] Lustre: Skipped 2 previous similar messages [ 2760.159251] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2760.335650] Lustre: server umount lustre-MDT0001 complete [ 2760.752514] LustreError: 98867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.785936] LustreError: 98867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2770.654204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.805875] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2770.817546] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2770.921019] Lustre: 98857:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2772.142351] LustreError: 102658:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2772.147936] LustreError: 102658:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2772.163924] LustreError: 98862:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2772.316333] LustreError: 102670:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.206.153@tcp: cfg command failed: rc = -17 [ 2772.329608] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2772.329608] [ 2776.680386] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 2777.073527] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2787.323910] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 08:16:35 (1775218595) [ 2805.095648] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 2823.282904] Lustre: 104713:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2846.336466] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2849.792684] Lustre: 105846:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2945.506619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2945.509382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.537469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2945.542525] Lustre: Skipped 8 previous similar messages [ 2948.881155] Lustre: server umount lustre-MDT0000 complete [ 2951.141610] LustreError: 98865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.160514] LustreError: 98865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2952.378249] LustreError: 98851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218763 with bad export cookie 18331327033053026249 [ 2952.379950] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2952.394041] LustreError: 98851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2952.810904] Lustre: server umount lustre-MDT0001 complete [ 2966.292295] Lustre: server umount lustre-OST0000 complete [ 2980.454534] Lustre: server umount lustre-OST0001 complete [ 2986.551477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2997.336046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3005.898340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3016.057422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3039.361223] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 3050.556396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.580050] Lustre: lustre-MDT0000: reset Object Index mappings [ 3051.254327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.268700] Lustre: Skipped 1 previous similar message [ 3057.406224] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3069.688713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.714806] Lustre: lustre-MDT0001: reset Object Index mappings [ 3077.010732] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3080.071155] Lustre: 110906:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3086.348247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.981250] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3095.549608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3095.580331] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3104.052778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.010576] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3107.037798] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3111.650221] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3122.347297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3127.125287] Lustre: 112757:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3192.466565] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:23:20 (1775219000) [ 3218.916317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3218.935235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.952901] Lustre: Skipped 3 previous similar messages [ 3218.965386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.980194] Lustre: Skipped 4 previous similar messages [ 3225.178139] Lustre: server umount lustre-MDT0000 complete [ 3228.141648] LustreError: 110532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.172167] LustreError: 110532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3229.388052] LustreError: 109773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219040 with bad export cookie 18331327033053073366 [ 3229.398625] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3229.424109] LustreError: 109773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3236.296689] Lustre: server umount lustre-MDT0001 complete [ 3247.445201] Lustre: server umount lustre-OST0000 complete [ 3258.248514] Lustre: server umount lustre-OST0001 complete [ 3266.134540] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 3278.191971] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 3326.106508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.323360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3326.342133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3326.397254] Lustre: lustre-MDT0000: new disk, initializing [ 3326.471361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.474561] Lustre: Skipped 3 previous similar messages [ 3326.495470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3331.604534] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3339.373787] Lustre: server umount lustre-MDT0000 complete [ 3349.924258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.219914] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3350.236180] Lustre: 116979:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3350.264702] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3354.977090] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3360.318465] Lustre: server umount lustre-MDT0000 complete [ 3369.965545] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:26:18 (1775219178) [ 3371.543570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3373.370499] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:26:22 (1775219182) [ 3393.278376] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 3404.542468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.400833] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3420.063138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.261925] Lustre: 118996:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3420.319352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3420.322381] Lustre: Skipped 1 previous similar message [ 3420.403250] Lustre: lustre-MDT0001: new disk, initializing [ 3420.579833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3420.593248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3425.641365] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3431.232840] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3438.963527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.388149] Lustre: lustre-OST0000: new disk, initializing [ 3439.400564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3440.809630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3440.828914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3440.956995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3446.650650] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3460.104477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.401962] Lustre: lustre-OST0001: new disk, initializing [ 3460.413151] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3461.573804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3461.587848] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3461.739241] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3467.812793] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 3477.853486] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3481.778639] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3865.445965] Lustre: 127792:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3865.453835] Lustre: 127792:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3865.464434] Lustre: 127792:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3865.474773] Lustre: 127792:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3865.481966] Lustre: 127792:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3865.487223] Lustre: 127792:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4120.643560] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:38:48 (1775219928) [ 4137.444309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.450062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.459247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.488699] Lustre: Skipped 8 previous similar messages [ 4137.515456] Lustre: Skipped 11 previous similar messages [ 4142.112736] Lustre: server umount lustre-MDT0000 complete [ 4142.581237] LustreError: 122215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.617930] LustreError: 122215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4146.081798] LustreError: 118989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219956 with bad export cookie 18331327033053143730 [ 4146.090639] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4146.093091] LustreError: 118989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4147.686231] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.694914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4147.702469] Lustre: Skipped 1 previous similar message [ 4147.714813] Lustre: Skipped 1 previous similar message [ 4152.607668] Lustre: server umount lustre-MDT0001 complete [ 4171.231487] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4171.403803] Lustre: server umount lustre-OST0000 complete [ 4181.677198] Lustre: server umount lustre-OST0001 complete [ 4188.454822] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 4198.421526] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 4232.128400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.386643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4232.425266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4232.502348] Lustre: lustre-MDT0000: new disk, initializing [ 4232.588573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.616098] Lustre: Skipped 5 previous similar messages [ 4232.655897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4236.494537] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 4245.379304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.478261] Lustre: 130805:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4245.552606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4245.559914] Lustre: Skipped 1 previous similar message [ 4245.643368] Lustre: lustre-MDT0001: new disk, initializing [ 4245.779809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4245.787228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4248.926147] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 4254.899809] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4257.943942] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4263.448124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4263.667254] Lustre: lustre-OST0000: new disk, initializing [ 4263.670786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4264.871409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4264.878204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4264.990488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4268.228949] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 4273.545181] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4280.166687] Lustre: lustre-MDD0000: changelog on [ 4282.247259] Lustre: lustre-MDD0001: changelog on [ 5484.454723] Lustre: lustre-MDD0001: changelog off [ 5485.313423] Lustre: 135604:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5485.732919] Lustre: lustre-MDD0000: changelog off [ 5488.096230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5488.100381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.108662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.112365] Lustre: Skipped 2 previous similar messages [ 5488.609978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.615352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5493.730176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5493.735353] Lustre: Skipped 2 previous similar messages [ 5494.193075] Lustre: server umount lustre-OST0000 complete [ 5498.848659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.850436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.857239] Lustre: Skipped 1 previous similar message [ 5502.200897] Lustre: server umount lustre-MDT0000 complete [ 5503.969534] LustreError: 131670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.976508] LustreError: 131670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5503.992273] LustreError: 140538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221314 with bad export cookie 18331327033053921451 [ 5503.995309] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5503.999033] LustreError: 140538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5504.177420] Lustre: server umount lustre-MDT0001 complete [ 5508.902824] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 5510.296835] Key type lgssc unregistered [ 5510.466568] LNet: 141472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5510.473241] LNetError: 141472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5510.483533] LNet: Removed LNI 192.168.206.153@tcp [ 5510.919111] Key type .llcrypt unregistered [ 5510.922745] Key type ._llcrypt unregistered [ 5519.491820] Key type ._llcrypt registered [ 5519.493444] Key type .llcrypt registered [ 5519.534878] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 5527.419360] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 5527.815245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5527.877330] alg: No test for adler32 (adler32-zlib) [ 5528.765572] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 5528.877851] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 5530.480132] Key type lgssc registered [ 5530.984437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5547.587766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5547.600303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.726792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5548.746454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5548.788545] Lustre: lustre-MDT0000: new disk, initializing [ 5548.824834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.838881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5550.464931] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5555.683631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.744712] Lustre: 144615:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5555.785594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5555.789361] Lustre: Skipped 1 previous similar message [ 5555.840727] Lustre: lustre-MDT0001: new disk, initializing [ 5555.895869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5555.921491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5555.935669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5557.745756] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5561.929863] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5563.663861] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5566.947918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5567.071531] Lustre: lustre-OST0000: new disk, initializing [ 5567.074726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5567.120155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.711928] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5573.819936] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5575.194121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5575.199826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5575.270987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5581.915650] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_module obdecho/obdecho [ 5582.916850] LustreError: 147226:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5582.923371] LustreError: 147226:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 5583.818625] LustreError: 147322:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5590.501398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.507129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5590.511143] Lustre: Skipped 1 previous similar message [ 5595.137367] Lustre: server umount lustre-MDT0000 complete [ 5597.187987] LustreError: 146362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221408 with bad export cookie 1416999834133300374 [ 5597.189690] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5597.193900] LustreError: 146362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5597.392278] Lustre: server umount lustre-MDT0001 complete [ 5609.741293] Lustre: server umount lustre-OST0000 complete [ 5619.795956] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 5623.840572] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 5637.787961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.888492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5637.905613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5637.947270] Lustre: lustre-MDT0000: new disk, initializing [ 5637.972775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.982396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5639.401876] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5644.031688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.081601] Lustre: 150999:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5644.109157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5644.112932] Lustre: Skipped 1 previous similar message [ 5644.152580] Lustre: lustre-MDT0001: new disk, initializing [ 5644.200131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5644.216441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5644.220779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5645.727458] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5649.378881] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5650.886418] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5653.858667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.972842] Lustre: lustre-OST0000: new disk, initializing [ 5653.976561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5654.025603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5656.452286] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5657.790987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5657.795912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5657.835031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5660.061403] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5661.748847] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5661.837770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5663.438552] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5663.527614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5668.323429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5668.324616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.328172] LustreError: Skipped 1 previous similar message [ 5668.329136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5668.335073] Lustre: Skipped 2 previous similar messages [ 5668.338555] Lustre: Skipped 2 previous similar messages [ 5670.424519] Lustre: server umount lustre-OST0000 complete [ 5673.440793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.450077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5673.453417] Lustre: Skipped 2 previous similar messages [ 5678.174634] Lustre: server umount lustre-MDT0000 complete [ 5678.560646] LustreError: 151008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.573157] LustreError: 151008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5679.817363] LustreError: 150989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221490 with bad export cookie 1416999834133301319 [ 5679.822155] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5679.823185] LustreError: 150989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5679.983200] Lustre: server umount lustre-MDT0001 complete [ 5681.590565] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:04:51 (1775221491) [ 5682.281123] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5683.019868] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:04:52 (1775221492) [ 5686.150066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.344712] LustreError: 154997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.375313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.858145] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5691.078732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.726212] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5694.719386] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5696.073906] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5699.156290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5701.771198] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5704.228700] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5720.112934] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.53@tcp (stopping) [ 5722.083048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.085644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5722.089021] Lustre: Skipped 3 previous similar messages [ 5723.968786] Lustre: server umount lustre-MDT0000 complete [ 5725.222096] LustreError: 154994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5725.644685] LustreError: 154978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221536 with bad export cookie 1416999834133302201 [ 5725.648385] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5725.650690] LustreError: 154978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5727.200766] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.207711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5727.211508] Lustre: Skipped 3 previous similar messages [ 5730.208427] LustreError: 154992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.220039] LustreError: 154992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5731.890268] Lustre: server umount lustre-MDT0001 complete [ 5734.767695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5740.041818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.172373] LustreError: 158518:0:(llog.c:1646:llog_backup()) MGC192.168.206.153@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5740.178305] Lustre: 158518:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.206.153@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5749.798577] LustreError: 158523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.806266] LustreError: 158523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5760.047763] LustreError: 158523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.066121] LustreError: 158523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5776.422413] LustreError: 158525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.433732] LustreError: 158525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 5801.439292] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5801.442133] Lustre: 158518:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.206.153@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5810.657510] LustreError: 158544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.668641] LustreError: 158544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 5841.897474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5841.900638] Lustre: Skipped 2 previous similar messages [ 5843.628849] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5845.989713] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5847.184234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.559638] LustreError: 142839:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e6a2fde4 x1861454490870912/t0(0) o250->MGC192.168.206.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5872.211028] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5874.420262] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5875.008273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5875.688787] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5876.342621] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5878.037925] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 5885.921268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5885.922821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.925583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5885.929949] Lustre: Skipped 1 previous similar message [ 5885.933900] Lustre: Skipped 4 previous similar messages [ 5889.086267] Lustre: server umount lustre-MDT0000 complete [ 5890.587700] LustreError: 159924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221701 with bad export cookie 1416999834133308648 [ 5890.590728] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.593077] LustreError: 159924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5890.725529] Lustre: server umount lustre-MDT0001 complete [ 5902.714600] Lustre: server umount lustre-OST0000 complete [ 5912.276750] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 5917.153937] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 5936.638168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5936.781222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5936.806117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5936.852234] Lustre: lustre-MDT0000: new disk, initializing [ 5936.902641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.906769] Lustre: Skipped 1 previous similar message [ 5936.916778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5938.693935] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5944.150744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.200645] Lustre: 163345:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5944.336983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5946.057186] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5949.925399] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5951.668156] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5954.926838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5957.785389] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 5959.690479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5959.736766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5961.451187] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5963.221689] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5963.316790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5965.067277] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5965.156784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5969.890604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5969.894320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.895839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5969.900504] Lustre: Skipped 2 previous similar messages [ 5972.142048] Lustre: server umount lustre-OST0000 complete [ 5975.010078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5975.013462] LustreError: Skipped 1 previous similar message [ 5988.319321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5988.407084] Lustre: server umount lustre-MDT0000 complete [ 5989.344991] LustreError: 163357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.352405] LustreError: 163357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 5990.161219] LustreError: 164122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221801 with bad export cookie 1416999834133325476 [ 5990.164383] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.167478] LustreError: 164122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5990.313827] Lustre: server umount lustre-MDT0001 complete [ 5994.378719] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:10:03 (1775221803) [ 5997.317440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.191381] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6003.067075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.030223] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6007.421630] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6009.037779] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6012.031711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.932643] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6017.282167] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6026.961995] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:10:36 (1775221836) [ 6028.768239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6028.771251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.776300] Lustre: Skipped 2 previous similar messages [ 6028.779367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6028.782102] Lustre: Skipped 7 previous similar messages [ 6034.268775] Lustre: server umount lustre-OST0000 complete [ 6050.271135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6050.354318] Lustre: server umount lustre-MDT0000 complete [ 6051.942278] LustreError: 167327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221862 with bad export cookie 1416999834133326358 [ 6051.942648] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.948815] LustreError: 167327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6052.084898] Lustre: server umount lustre-MDT0001 complete [ 6056.127862] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 6057.298318] Key type lgssc unregistered [ 6057.434331] LNet: 170720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6057.438219] LNetError: 170720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6057.453293] LNet: Removed LNI 192.168.206.153@tcp [ 6057.785722] Key type .llcrypt unregistered [ 6057.787402] Key type ._llcrypt unregistered [ 6065.758137] Key type ._llcrypt registered [ 6065.759802] Key type .llcrypt registered [ 6065.800774] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6066.231731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6066.245817] alg: No test for adler32 (adler32-zlib) [ 6067.117770] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6067.220255] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 6068.807181] Key type lgssc registered [ 6069.281610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6072.507831] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6072.521264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6087.967300] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6093.087314] LustreError: 171922:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.153@tcp: failed processing log, type 4: rc = -110 [ 6098.207501] LustreError: 171922:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6098.212650] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6098.215400] LustreError: MGC192.168.206.153@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6098.220555] LustreError: 171922:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6098.224313] LustreError: 171922:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 6098.227506] LustreError: 171922:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 6098.250335] Lustre: server umount lustre-OST0000 complete [ 6098.252487] LustreError: 171922:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6101.497425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.657133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6104.030857] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6107.163203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.324441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6108.758411] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6110.731469] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6112.147724] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6114.814032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6114.895313] LustreError: 174008:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6114.900825] Lustre: 174008:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.206.153@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6114.966981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.145197] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6119.155839] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6119.881056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.882472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6119.884983] Lustre: Skipped 2 previous similar messages [ 6119.887409] Lustre: Skipped 1 previous similar message [ 6125.025432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6125.031181] Lustre: Skipped 3 previous similar messages [ 6130.145148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6130.147522] Lustre: Skipped 2 previous similar messages [ 6134.239138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6134.305847] Lustre: server umount lustre-MDT0000 complete [ 6135.264304] LustreError: 172422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.269525] LustreError: 172422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6135.653403] LustreError: 172401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221946 with bad export cookie 1607182221988840540 [ 6135.656361] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6135.659302] LustreError: 172401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6135.777880] Lustre: server umount lustre-MDT0001 complete [ 6146.340335] Lustre: server umount lustre-OST0000 complete [ 6149.705547] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 6153.868988] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6167.908071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.009879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6168.025491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6168.066589] Lustre: lustre-MDT0000: new disk, initializing [ 6168.098316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.107273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6169.602742] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6174.572025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.616854] Lustre: 177385:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6174.642218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6174.645053] Lustre: Skipped 1 previous similar message [ 6174.684800] Lustre: lustre-MDT0001: new disk, initializing [ 6174.731825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6174.749909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6174.754875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6176.336162] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6179.815073] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6181.279434] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6183.833346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6183.951197] Lustre: lustre-OST0000: new disk, initializing [ 6183.953770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6183.983916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6186.012693] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6189.373678] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6190.823693] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6193.144346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6193.148300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6193.172039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6193.970707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6195.602793] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6195.699860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6198.242178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.244460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6198.246765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6198.248180] LustreError: Skipped 1 previous similar message [ 6198.250546] Lustre: Skipped 1 previous similar message [ 6202.509630] Lustre: server umount lustre-OST0000 complete [ 6205.407776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6205.410754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6205.415522] Lustre: Skipped 1 previous similar message [ 6205.417553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6208.481736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.600317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6213.602795] Lustre: Skipped 3 previous similar messages [ 6218.207149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6218.275557] Lustre: server umount lustre-MDT0000 complete [ 6218.720401] LustreError: 178248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.726896] LustreError: 178248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6219.692453] LustreError: 177378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222030 with bad export cookie 1607182221988841240 [ 6219.696370] LustreError: 177378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6219.696405] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6219.821202] Lustre: server umount lustre-MDT0001 complete [ 6223.007963] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:13:52 (1775222032) [ 6229.503885] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 6230.590904] Key type lgssc unregistered [ 6230.728378] LNet: 181930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6230.731719] LNetError: 181930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6230.743349] LNet: Removed LNI 192.168.206.153@tcp [ 6231.059502] Key type .llcrypt unregistered [ 6231.060759] Key type ._llcrypt unregistered [ 6239.347818] Key type ._llcrypt registered [ 6239.349232] Key type .llcrypt registered [ 6239.398901] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6239.799344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6239.805278] alg: No test for adler32 (adler32-zlib) [ 6240.658855] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6240.749973] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 6242.335139] Key type lgssc registered [ 6242.752165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6245.690901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6245.703451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.879767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.384386] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6251.819826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.976839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6253.572074] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6256.081631] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6257.815095] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6261.195699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6261.310841] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6261.313516] LustreError: 184756:0:(llog.c:1687:llog_backup()) MGC192.168.206.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6261.317594] Lustre: 184756:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.206.153@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6261.365057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6263.718549] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6265.747560] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6266.848806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6266.851218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6266.855388] Lustre: Skipped 2 previous similar messages [ 6266.858886] Lustre: Skipped 1 previous similar message [ 6271.968545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6271.971466] Lustre: Skipped 3 previous similar messages [ 6272.670497] Lustre: server umount lustre-MDT0000 complete [ 6274.227512] LustreError: 183104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222085 with bad export cookie 3048757970321863879 [ 6274.230452] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6274.233377] LustreError: 183104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6274.357174] Lustre: server umount lustre-MDT0001 complete [ 6286.101068] Lustre: server umount lustre-OST0000 complete [ 6289.818351] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 6294.255225] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6307.364747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.464582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6307.480317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6307.521513] Lustre: lustre-MDT0000: new disk, initializing [ 6307.546881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.555175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6309.012692] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6313.382407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.432776] Lustre: 188130:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6313.457537] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6313.459076] Lustre: Skipped 1 previous similar message [ 6313.460770] LustreError: 188894:0:(llog.c:1687:llog_backup()) MGC192.168.206.153@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6313.466148] LustreError: 188894:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6313.470100] Lustre: 188894:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.206.153@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6313.481121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6313.483628] Lustre: Skipped 1 previous similar message [ 6313.519431] Lustre: lustre-MDT0001: new disk, initializing [ 6313.558290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6313.572749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6313.577378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6315.103744] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6318.365706] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6319.663433] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6322.204275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6322.305332] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6322.307870] Lustre: Skipped 1 previous similar message [ 6322.310331] LustreError: 189848:0:(llog.c:1687:llog_backup()) MGC192.168.206.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6322.315395] LustreError: 189848:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6322.318242] Lustre: 189848:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.206.153@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6322.322713] Lustre: 189848:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6322.333772] Lustre: lustre-OST0000: new disk, initializing [ 6322.336255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6322.367729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.468294] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6327.817179] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6329.302638] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6332.410188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6332.414586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6332.448258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6333.447312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6334.720191] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6334.789159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6337.505396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6337.505560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.507262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.514286] Lustre: Skipped 1 previous similar message [ 6341.522552] Lustre: server umount lustre-OST0000 complete [ 6344.159688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6344.162704] LustreError: Skipped 1 previous similar message [ 6344.164415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.170981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6344.173323] Lustre: Skipped 1 previous similar message [ 6347.744932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.147777] Lustre: server umount lustre-MDT0000 complete [ 6350.440968] LustreError: 188909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222161 with bad export cookie 3048757970321864572 [ 6350.444349] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.445597] LustreError: 188909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6350.569818] Lustre: server umount lustre-MDT0001 complete [ 6353.531680] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:16:03 (1775222163) [ 6358.576432] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6361.773065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.906714] LustreError: 192853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.916197] LustreError: 192853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6361.920444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.223944] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6366.070015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.615433] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6368.688414] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6370.862706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6372.977544] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6376.092118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6376.162299] Lustre: lustre-OST0001: new disk, initializing [ 6376.164513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6377.225468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6377.229722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6377.260267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6378.155593] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6383.644668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6390.357382] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6395.985435] Lustre: DEBUG MARKER: ADD OST3 [ 6397.711883] loop: module loaded [ 6397.728719] loop0: detected capacity change from 0 to 204800000 [ 6397.802793] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6397.807731] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6397.812939] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6399.763028] loop0: detected capacity change from 0 to 204800000 [ 6401.560710] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6401.636492] Lustre: lustre-OST0003: new disk, initializing [ 6401.638819] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6401.676523] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6401.679621] Lustre: Skipped 3 previous similar messages [ 6402.743521] Lustre: DEBUG MARKER: STOP OST3 [ 6403.095864] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6403.141742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6403.147354] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6403.160104] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6403.476802] Lustre: Failing over lustre-OST0003 [ 6403.524992] Lustre: server umount lustre-OST0003 complete [ 6403.808920] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6405.151291] LustreError: 197130:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6408.102173] Lustre: DEBUG MARKER: START OST3 again [ 6408.166934] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.168085] LustreError: 194302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6408.173027] Lustre: Skipped 1 previous similar message [ 6408.174581] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6410.412232] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6410.506033] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6412.011485] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6412.321390] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6412.407257] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6412.407691] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6425.805071] Lustre: Failing over lustre-OST0003 [ 6425.843602] Lustre: server umount lustre-OST0003 complete [ 6426.083808] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.086494] LustreError: 194302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.088200] Lustre: Skipped 1 previous similar message [ 6426.096295] LustreError: 194302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6431.204561] LustreError: 194488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.210115] LustreError: 194488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6436.329901] LustreError: 194305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6436.334677] LustreError: 194305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6441.448043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6441.449832] Lustre: Skipped 4 previous similar messages [ 6444.447409] Lustre: server umount lustre-MDT0000 complete [ 6445.695100] LustreError: 193953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222256 with bad export cookie 3048757970321865440 [ 6445.698845] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6445.699112] LustreError: 193953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6446.559947] LustreError: 195884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.560517] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.567174] LustreError: 195884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6446.572972] Lustre: Skipped 5 previous similar messages [ 6450.592246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6450.594255] Lustre: Skipped 3 previous similar messages [ 6452.016405] Lustre: server umount lustre-MDT0001 complete [ 6453.287182] Lustre: server umount lustre-OST0000 complete [ 6454.521381] Lustre: server umount lustre-OST0001 complete [ 6456.611784] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 6459.228968] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6473.112685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.211750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6473.226139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6473.265151] Lustre: lustre-MDT0000: new disk, initializing [ 6473.297224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6473.300016] Lustre: Skipped 1 previous similar message [ 6473.308452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6474.865886] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6479.384291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.435548] Lustre: 202172:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6479.549808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6481.031442] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6484.275629] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6485.692931] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6488.161621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6489.402502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6489.405813] Lustre: Skipped 1 previous similar message [ 6489.407723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6489.443558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6490.309713] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6493.588833] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6495.000114] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6495.074805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6496.440979] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6496.526749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6499.812101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6499.813070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6499.814552] LustreError: Skipped 1 previous similar message [ 6499.819598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6499.822806] Lustre: Skipped 1 previous similar message [ 6503.318874] Lustre: server umount lustre-OST0000 complete [ 6510.812486] Lustre: server umount lustre-MDT0000 complete [ 6512.233707] LustreError: 202950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222323 with bad export cookie 3048757970321867694 [ 6512.236738] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6512.239469] LustreError: 202950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6512.369129] Lustre: server umount lustre-MDT0001 complete [ 6513.716084] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:18:43 (1775222323) [ 6519.111851] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 6522.300250] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6534.405798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.511525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6534.525749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6534.527612] Lustre: Skipped 3 previous similar messages [ 6534.566153] Lustre: lustre-MDT0000: new disk, initializing [ 6534.568339] Lustre: Skipped 2 previous similar messages [ 6534.600036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6536.010562] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6540.154595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.192926] Lustre: 208335:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6540.284128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6541.514916] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6544.489874] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6545.574264] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6547.748670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6549.025071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6549.737472] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6552.874451] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6554.189094] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6554.262889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6555.542255] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6555.623416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6559.201988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6559.202804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6559.204032] LustreError: Skipped 1 previous similar message [ 6559.205918] Lustre: Skipped 4 previous similar messages [ 6562.382088] Lustre: server umount lustre-OST0000 complete [ 6564.321726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.327332] Lustre: Skipped 6 previous similar messages [ 6569.751821] Lustre: server umount lustre-MDT0000 complete [ 6570.987375] LustreError: 209114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222381 with bad export cookie 3048757970321868576 [ 6570.991281] LustreError: 209114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6570.991424] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6571.104577] Lustre: server umount lustre-MDT0001 complete [ 6573.973789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.132726] LustreError: 212145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.140157] LustreError: 212145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6575.528610] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6578.486520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.933543] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6581.843199] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6583.072264] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6585.313936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6587.348075] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6589.012280] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6609.380155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6609.382937] LustreError: Skipped 1 previous similar message [ 6611.855336] Lustre: server umount lustre-OST0000 complete [ 6619.280734] Lustre: server umount lustre-MDT0000 complete [ 6619.616140] LustreError: 212146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.621389] LustreError: 212146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6620.488080] LustreError: 212125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222431 with bad export cookie 3048757970321869458 [ 6620.490666] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.491931] LustreError: 212125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6620.598547] Lustre: server umount lustre-MDT0001 complete [ 6721.641310] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 6722.647762] Key type lgssc unregistered [ 6722.776434] LNet: 215472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6722.779768] LNetError: 215472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6722.787364] LNet: Removed LNI 192.168.206.153@tcp [ 6723.091143] Key type .llcrypt unregistered [ 6723.092399] Key type ._llcrypt unregistered [ 6731.005941] Key type ._llcrypt registered [ 6731.007346] Key type .llcrypt registered [ 6731.043475] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 6736.989487] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6737.313463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6737.324025] alg: No test for adler32 (adler32-zlib) [ 6738.177812] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6738.256740] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 6739.839154] Key type lgssc registered [ 6740.240991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6751.185689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6751.192907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.271566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6752.281698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6752.315473] Lustre: lustre-MDT0000: new disk, initializing [ 6752.336217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.341714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6753.507440] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6757.409545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.443977] Lustre: 218759:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6757.462847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6757.464328] Lustre: Skipped 1 previous similar message [ 6757.496288] Lustre: lustre-MDT0001: new disk, initializing [ 6757.521542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6757.531773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6757.535184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6758.771511] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6761.746741] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6762.880886] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6765.036674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.142977] Lustre: lustre-OST0000: new disk, initializing [ 6765.145750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6765.178423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.965669] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6767.666347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6767.670191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6767.698250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6769.878604] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6771.138079] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6771.210131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6772.430158] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6772.494768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6775.115289] Lustre: server umount lustre-OST0000 complete [ 6777.824707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.825138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6777.828145] Lustre: Skipped 1 previous similar message [ 6782.944023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6782.946271] Lustre: Skipped 2 previous similar messages [ 6787.065331] Lustre: server umount lustre-MDT0000 complete [ 6788.209874] LustreError: 219537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222599 with bad export cookie 15228404685424495803 [ 6788.212142] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6788.212718] LustreError: 219537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6798.859581] Lustre: server umount lustre-MDT0001 complete [ 6804.268137] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6807.584025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.715171] LustreError: 223269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.721131] LustreError: 223269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6807.731293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6808.939699] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6811.623831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.994663] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6813.914761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6815.981880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6816.098450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.100624] Lustre: Skipped 1 previous similar message [ 6817.873251] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6820.557238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6820.613834] Lustre: lustre-OST0001: new disk, initializing [ 6820.615426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6822.484209] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6822.721859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6822.725067] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6822.757590] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6827.799872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6833.414593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6842.054592] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:24:11 (1775222651) [ 6843.362329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6843.363834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6843.367444] Lustre: Skipped 3 previous similar messages [ 6843.369014] Lustre: Skipped 2 previous similar messages [ 6848.481646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6848.484301] Lustre: Skipped 2 previous similar messages [ 6848.924242] Lustre: server umount lustre-MDT0000 complete [ 6850.102124] LustreError: 226335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222660 with bad export cookie 15228404685424496671 [ 6850.104779] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6850.105059] LustreError: 226335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6850.214914] Lustre: server umount lustre-MDT0001 complete [ 6861.821412] Lustre: server umount lustre-OST0000 complete [ 6872.277887] Lustre: server umount lustre-OST0001 complete [ 6874.306707] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 6877.252876] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 6888.697514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.787320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6888.797758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6888.831874] Lustre: lustre-MDT0000: new disk, initializing [ 6888.853701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6888.855615] Lustre: Skipped 1 previous similar message [ 6888.861016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6890.037391] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6893.955907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.991863] Lustre: 229417:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6894.084337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6895.272024] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6898.287785] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6899.394987] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6901.436256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6901.531033] Lustre: lustre-OST0000: new disk, initializing [ 6901.532590] Lustre: Skipped 1 previous similar message [ 6901.534153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6901.535849] Lustre: Skipped 2 previous similar messages [ 6903.161742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6903.165518] Lustre: Skipped 1 previous similar message [ 6903.167912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6903.200864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6903.351542] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6906.307662] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6907.630093] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6907.701577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6908.952410] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6909.018728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6913.505096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6913.505709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6913.508316] LustreError: Skipped 1 previous similar message [ 6913.512236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6913.515499] Lustre: Skipped 1 previous similar message [ 6915.724516] Lustre: server umount lustre-OST0000 complete [ 6918.624493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.628442] Lustre: Skipped 2 previous similar messages [ 6923.028416] Lustre: server umount lustre-MDT0000 complete [ 6923.743984] LustreError: 229428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6923.750747] LustreError: 229428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6924.184203] LustreError: 229409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222735 with bad export cookie 15228404685424498771 [ 6924.186284] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.187311] LustreError: 229409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6924.298422] Lustre: server umount lustre-MDT0001 complete [ 6926.904452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.045965] LustreError: 233225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.067538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.070423] Lustre: Skipped 2 previous similar messages [ 6928.254267] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6930.896289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.193582] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6933.825242] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6934.929251] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6937.002944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.057479] LustreError: 234816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6938.817286] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 6940.359953] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7084.001240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7084.001963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7084.002685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7084.002690] Lustre: Skipped 2 previous similar messages [ 7084.004855] Lustre: Skipped 1 previous similar message [ 7098.335109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7098.380827] Lustre: server umount lustre-OST0000 complete [ 7099.871672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7099.873984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.479820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7104.481909] Lustre: Skipped 7 previous similar messages [ 7105.614835] Lustre: server umount lustre-MDT0000 complete [ 7106.803253] LustreError: 233205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222917 with bad export cookie 15228404685424499653 [ 7106.806405] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7106.808557] LustreError: 233205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7106.947932] Lustre: server umount lustre-MDT0001 complete [ 7205.054134] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 7206.126122] Key type lgssc unregistered [ 7206.256363] LNet: 236478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7206.260176] LNetError: 236478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7206.266320] LNet: Removed LNI 192.168.206.153@tcp [ 7206.567134] Key type .llcrypt unregistered [ 7206.568454] Key type ._llcrypt unregistered [ 7214.530022] Key type ._llcrypt registered [ 7214.531462] Key type .llcrypt registered [ 7214.570162] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 7220.856766] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7221.322878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7221.340040] alg: No test for adler32 (adler32-zlib) [ 7222.204529] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7222.297544] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 7223.879112] Key type lgssc registered [ 7224.242578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7238.646179] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7242.229436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7242.235323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.332828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7243.346878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7243.386721] Lustre: lustre-MDT0000: new disk, initializing [ 7243.411989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.419665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7244.793485] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7249.033962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.077682] Lustre: 240484:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7249.099620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7249.102362] Lustre: Skipped 1 previous similar message [ 7249.138072] Lustre: lustre-MDT0001: new disk, initializing [ 7249.168778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7249.179900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7249.183097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7250.595378] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7252.774643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7254.719835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.811070] Lustre: lustre-OST0000: new disk, initializing [ 7254.812618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7254.837602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.462789] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7256.689635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7256.693625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7256.724172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7260.564574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7260.633119] Lustre: lustre-OST0001: new disk, initializing [ 7260.635143] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7260.675763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7262.514412] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7265.790307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7265.794193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7265.821139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7267.991179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7271.645381] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7280.096034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7280.098311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.102091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7281.121066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7281.123382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.127707] Lustre: Skipped 1 previous similar message [ 7285.667765] Lustre: server umount lustre-MDT0000 complete [ 7286.878441] LustreError: 243684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223097 with bad export cookie 266301532963196236 [ 7286.882316] LustreError: 243684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7286.882586] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7287.007684] Lustre: server umount lustre-MDT0001 complete [ 7298.566543] Lustre: server umount lustre-OST0000 complete [ 7310.101142] Lustre: server umount lustre-OST0001 complete [ 7314.139600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7316.885820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7319.352421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7322.240066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7327.904436] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7330.918720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.924980] Lustre: lustre-MDT0000: reset Object Index mappings [ 7331.045075] LustreError: 247899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.050867] LustreError: 247899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7331.063269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7332.187542] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7334.939163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.946739] Lustre: lustre-MDT0001: reset Object Index mappings [ 7336.408291] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7337.363066] Lustre: 249007:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7339.581926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7339.700461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.703021] Lustre: Skipped 1 previous similar message [ 7340.706028] LustreError: 249361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.710053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7341.573598] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7344.540148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7345.643188] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7346.535765] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7349.682925] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7350.933321] Lustre: 250848:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7360.995185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7360.997558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7361.001854] Lustre: Skipped 3 previous similar messages [ 7361.003884] Lustre: Skipped 4 previous similar messages [ 7361.006967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7363.223324] Lustre: server umount lustre-MDT0000 complete [ 7364.449028] LustreError: 247879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223175 with bad export cookie 266301532963203012 [ 7364.451426] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7364.454149] LustreError: 247879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7364.575272] Lustre: server umount lustre-MDT0001 complete [ 7375.294884] Lustre: server umount lustre-OST0000 complete [ 7386.779476] Lustre: server umount lustre-OST0001 complete [ 7393.119922] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:33:22 (1775223202) [ 7398.287967] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 7401.530568] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7413.677704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.768573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7413.781985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7413.821363] Lustre: lustre-MDT0000: new disk, initializing [ 7413.848331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7413.851562] Lustre: Skipped 1 previous similar message [ 7413.857541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7415.167197] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7419.311267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.355203] Lustre: 255058:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7419.461327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7420.807067] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7423.863690] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7425.002511] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7427.223989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7427.324443] Lustre: lustre-OST0000: new disk, initializing [ 7427.326200] Lustre: Skipped 1 previous similar message [ 7427.328414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7427.331277] Lustre: Skipped 2 previous similar messages [ 7428.471517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7428.475283] Lustre: Skipped 1 previous similar message [ 7428.477694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7428.513863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7429.204963] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7432.317115] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7433.645060] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7433.716992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7435.032184] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7435.103824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7438.816870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7438.819195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.820146] LustreError: Skipped 1 previous similar message [ 7438.824345] Lustre: Skipped 1 previous similar message [ 7438.825022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7438.829472] Lustre: Skipped 3 previous similar messages [ 7441.803719] Lustre: server umount lustre-OST0000 complete [ 7443.936721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.937291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.940530] Lustre: Skipped 2 previous similar messages [ 7449.170933] Lustre: server umount lustre-MDT0000 complete [ 7450.405125] LustreError: 255052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223261 with bad export cookie 266301532963205105 [ 7450.408129] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.408229] LustreError: 255052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7450.532781] Lustre: server umount lustre-MDT0001 complete [ 7456.317217] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7459.692996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.826979] LustreError: 259564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7459.842527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7459.844619] Lustre: Skipped 2 previous similar messages [ 7461.155890] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7464.019021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.456893] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7466.424952] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7468.591968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7470.509966] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7473.335883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7473.399274] Lustre: lustre-OST0001: new disk, initializing [ 7473.400979] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7475.200471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7475.205762] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7475.238945] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7475.330284] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7479.758437] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7480.982833] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7487.647248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7487.653776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7490.016164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7490.018361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.022390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7490.024282] Lustre: Skipped 3 previous similar messages [ 7495.515715] Lustre: server umount lustre-MDT0000 complete [ 7495.648859] LustreError: 259564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7495.657193] LustreError: 259564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7496.762225] LustreError: 261820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223307 with bad export cookie 266301532963205987 [ 7496.766345] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7496.767241] LustreError: 261820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7496.885949] Lustre: server umount lustre-MDT0001 complete [ 7508.997546] Lustre: server umount lustre-OST0000 complete [ 7519.384255] Lustre: server umount lustre-OST0001 complete [ 7524.082703] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7527.090860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.224927] LustreError: 264840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7527.243741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.245725] Lustre: Skipped 3 previous similar messages [ 7528.520254] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7531.352310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.796130] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7533.768961] Lustre: 265947:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7536.044322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7537.186215] LustreError: 266298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.192097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7538.078742] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7540.969717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7542.121960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7542.810467] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7545.697621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7546.796245] Lustre: 267796:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7553.555041] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7553.558963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7556.542794] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:36:06 (1775223366) [ 7572.448036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7572.451395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7572.455870] Lustre: Skipped 3 previous similar messages [ 7572.458585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7572.460658] Lustre: Skipped 3 previous similar messages [ 7574.170061] Lustre: server umount lustre-MDT0000 complete [ 7575.387537] LustreError: 267798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223386 with bad export cookie 266301532963212301 [ 7575.388757] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.392348] LustreError: 267798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7578.079724] LustreError: 266320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.083971] LustreError: 266320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7581.547861] Lustre: server umount lustre-MDT0001 complete [ 7588.925991] Lustre: server umount lustre-OST0000 complete [ 7596.249789] Lustre: server umount lustre-OST0001 complete [ 7598.372018] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 7601.243528] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7614.735491] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7617.996722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7618.106835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7618.120545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7618.164746] Lustre: lustre-MDT0000: new disk, initializing [ 7618.198413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7619.438869] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7623.207598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.244886] Lustre: 271976:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7623.333661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7624.455181] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7628.537258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7629.842063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7630.336829] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7634.193029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7635.357933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7636.039192] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7640.242685] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7649.945751] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 09:37:39 (1775223459) [ 7650.785746] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.786718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7650.789938] Lustre: Skipped 7 previous similar messages [ 7650.793951] Lustre: Skipped 10 previous similar messages [ 7656.727321] Lustre: server umount lustre-MDT0000 complete [ 7657.843979] LustreError: 275170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223468 with bad export cookie 266301532963220169 [ 7657.847470] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7657.847907] LustreError: 275170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7657.953056] Lustre: server umount lustre-MDT0001 complete [ 7668.232392] Lustre: server umount lustre-OST0000 complete [ 7679.714839] Lustre: server umount lustre-OST0001 complete [ 7685.701847] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 7688.998737] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7701.290994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.376467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7701.382459] Lustre: Skipped 2 previous similar messages [ 7701.393520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7701.396097] Lustre: Skipped 4 previous similar messages [ 7701.436668] Lustre: lustre-MDT0000: new disk, initializing [ 7701.439114] Lustre: Skipped 3 previous similar messages [ 7701.463793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.465913] Lustre: Skipped 7 previous similar messages [ 7701.471596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7701.475033] Lustre: Skipped 3 previous similar messages [ 7702.765785] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7706.947603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.992324] Lustre: 279001:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7707.088224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7707.090484] Lustre: Skipped 2 previous similar messages [ 7708.386482] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7711.481878] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7712.688829] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7714.967337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.969893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7717.038226] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7720.120083] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7720.847292] Lustre: Mounted lustre-client [ 7721.157960] LustreError: 281498:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff921b01800800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7721.184048] Lustre: Unmounted lustre-client [ 7726.050025] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.050642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.053689] Lustre: Skipped 2 previous similar messages [ 7726.056797] Lustre: Skipped 5 previous similar messages [ 7730.196489] Lustre: server umount lustre-OST0000 complete [ 7732.704057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7737.638123] Lustre: server umount lustre-MDT0000 complete [ 7738.924410] LustreError: 278994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223549 with bad export cookie 266301532963221695 [ 7738.927588] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7738.929027] LustreError: 278994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7739.050760] Lustre: server umount lustre-MDT0001 complete [ 7742.642965] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 7743.680961] Key type lgssc unregistered [ 7776.287171] LNet: 1 local NIs in recovery (showing 1): 192.168.206.153@tcp [ 7776.607442] LNet: 282775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7776.610231] LNetError: 282775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7853.413292] LNet: Removed LNI 192.168.206.153@tcp [ 7853.718094] Key type .llcrypt unregistered [ 7853.719282] Key type ._llcrypt unregistered [ 7859.526975] Key type ._llcrypt registered [ 7859.528083] Key type .llcrypt registered [ 7859.565928] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 7865.953318] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7866.440195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7866.477139] alg: No test for adler32 (adler32-zlib) [ 7867.324661] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7867.404782] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 7868.983151] Key type lgssc registered [ 7869.363636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7879.258892] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:41:28 (1775223688) [ 7881.520089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7881.527632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.619540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7882.631071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7882.671043] Lustre: lustre-MDT0000: new disk, initializing [ 7882.696957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7882.705020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7884.083541] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7888.177070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.215247] Lustre: 285936:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7888.235300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7888.236885] Lustre: Skipped 1 previous similar message [ 7888.270944] Lustre: lustre-MDT0001: new disk, initializing [ 7888.304786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7888.317099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7888.320375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7889.673076] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7892.819830] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7894.081701] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7896.425739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7896.534262] Lustre: lustre-OST0000: new disk, initializing [ 7896.537117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7896.567838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.170121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7898.173279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7898.210492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7898.489777] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7901.579233] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7908.321755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7908.324265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.325074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7908.330289] Lustre: Skipped 1 previous similar message [ 7911.502677] Lustre: server umount lustre-OST0000 complete [ 7913.441911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.442485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7913.445815] Lustre: Skipped 1 previous similar message [ 7913.447612] Lustre: Skipped 1 previous similar message [ 7918.559967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7918.562599] Lustre: Skipped 2 previous similar messages [ 7918.813033] Lustre: server umount lustre-MDT0000 complete [ 7919.924874] LustreError: 288286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223730 with bad export cookie 11724502151710707104 [ 7919.927213] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7919.929145] LustreError: 288286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7920.023173] Lustre: server umount lustre-MDT0001 complete [ 7923.307480] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 7924.199579] Key type lgssc unregistered [ 7924.309332] LNet: 289307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7924.312289] LNetError: 289307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7924.326326] LNet: Removed LNI 192.168.206.153@tcp [ 7924.568696] Key type .llcrypt unregistered [ 7924.569890] Key type ._llcrypt unregistered [ 7933.398653] Key type ._llcrypt registered [ 7933.400251] Key type .llcrypt registered [ 7933.436207] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 7933.804116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7933.810344] alg: No test for adler32 (adler32-zlib) [ 7934.652608] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7934.728162] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 7936.303156] Key type lgssc registered [ 7936.641511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7938.957533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7938.964693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.100060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.273371] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7943.730607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.830823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7944.984771] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7946.830545] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7948.012464] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7950.266539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.307725] LustreError: 292276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.312213] LustreError: 292276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7950.370967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.315773] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7954.130959] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7957.523244] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7957.975413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7959.519596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7959.521408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.524231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7960.621483] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.53@tcp (stopping) [ 7961.240231] Lustre: server umount lustre-MDT0000 complete [ 7962.438094] LustreError: 290665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223773 with bad export cookie 11941119964995200447 [ 7962.441419] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7962.443026] LustreError: 290665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7963.616032] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.616583] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7963.622389] Lustre: Skipped 1 previous similar message [ 7963.627290] LustreError: 290679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.732771] LustreError: 290678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.739611] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.53@tcp (stopping) [ 7968.628211] Lustre: server umount lustre-MDT0001 complete [ 7970.859598] Lustre: lustre-OST0000: Not available for connect from 192.168.206.53@tcp (stopping) [ 7970.861348] Lustre: Skipped 1 previous similar message [ 7981.093150] Lustre: lustre-OST0000: Not available for connect from 192.168.206.53@tcp (stopping) [ 7981.096958] Lustre: Skipped 1 previous similar message [ 7984.095086] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7984.160420] Lustre: server umount lustre-OST0000 complete [ 7987.050654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.405358] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7989.371848] Lustre: 294409:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.153@tcp [ 7990.098638] Lustre: server umount MGS complete [ 7992.800167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.940748] LustreError: 294980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7992.946221] LustreError: 294980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7992.959866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.106807] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 7996.750362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.063640] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8000.639391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8000.733466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.736377] Lustre: Skipped 1 previous similar message [ 8002.484611] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8004.025757] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8011.627957] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 8012.704397] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8017.259158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 8018.399927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8018.402673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.407374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.501148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8022.503907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.467449] Lustre: server umount lustre-OST0000 complete [ 8027.616732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8027.617137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.625567] Lustre: Skipped 1 previous similar message [ 8031.955561] Lustre: server umount lustre-MDT0000 complete [ 8032.736290] LustreError: 294981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8032.740718] LustreError: 294981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8033.241553] LustreError: 295005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223844 with bad export cookie 11941119964995201273 [ 8033.244526] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8033.248056] LustreError: 295005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8033.364932] Lustre: server umount lustre-MDT0001 complete [ 8037.106474] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8038.184567] Key type lgssc unregistered [ 8038.310372] LNet: 298154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8038.314201] LNetError: 298154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8038.324450] LNet: Removed LNI 192.168.206.153@tcp [ 8038.652937] Key type .llcrypt unregistered [ 8038.654640] Key type ._llcrypt unregistered [ 8044.987371] Key type ._llcrypt registered [ 8044.988396] Key type .llcrypt registered [ 8045.027186] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 8051.541144] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 8051.998477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8052.063349] alg: No test for adler32 (adler32-zlib) [ 8052.937787] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8053.038326] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 8054.623130] Key type lgssc registered [ 8055.006718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8066.368251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8066.374361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.466536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8067.477667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8067.518945] Lustre: lustre-MDT0000: new disk, initializing [ 8067.543198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8067.550623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8068.871702] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8072.984132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8073.023830] Lustre: 301249:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8073.046349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8073.049488] Lustre: Skipped 1 previous similar message [ 8073.087412] Lustre: lustre-MDT0001: new disk, initializing [ 8073.118251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8073.129990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8073.132991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8074.386956] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8077.488544] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8078.673256] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8080.837871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.937986] Lustre: lustre-OST0000: new disk, initializing [ 8080.940076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8080.966559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8082.797830] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8083.574888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8083.578105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8083.606966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8085.897414] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8087.307359] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8087.382244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8088.703643] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8088.768863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8091.473077] Lustre: server umount lustre-OST0000 complete [ 8093.663941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8093.666736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.670582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.176302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.176908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.296169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8099.298214] Lustre: Skipped 2 previous similar messages [ 8104.416263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8104.418611] Lustre: Skipped 1 previous similar message [ 8106.975120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8107.040118] Lustre: server umount lustre-MDT0000 complete [ 8108.259242] LustreError: 301241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223919 with bad export cookie 14540510493405560902 [ 8108.263253] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8108.264875] LustreError: 301241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8118.808589] Lustre: server umount lustre-MDT0001 complete [ 8121.628316] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:45:31 (1775223931) [ 8126.778695] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 8130.031157] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 8141.556840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.647759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8141.657958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8141.692944] Lustre: lustre-MDT0000: new disk, initializing [ 8141.716018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8141.723704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8143.033933] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8147.165216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.211123] Lustre: 307555:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8147.233839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8147.235757] Lustre: Skipped 1 previous similar message [ 8147.275304] Lustre: lustre-MDT0001: new disk, initializing [ 8147.312363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8147.327333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8147.330836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8148.714015] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8151.868445] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8153.056692] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8155.323030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8155.448397] Lustre: lustre-OST0000: new disk, initializing [ 8155.450807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8155.482833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.939958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8155.943784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8155.977337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8157.360886] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8160.491992] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8171.489446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.490461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.491846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.492963] LustreError: Skipped 1 previous similar message [ 8171.498093] Lustre: Skipped 1 previous similar message [ 8175.888272] Lustre: server umount lustre-OST0000 complete [ 8177.086167] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8177.088451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.728206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8181.728726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.730937] Lustre: Skipped 2 previous similar messages [ 8183.256581] Lustre: server umount lustre-MDT0000 complete [ 8184.530242] LustreError: 307548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223995 with bad export cookie 14540510493405561770 [ 8184.533132] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8184.537060] LustreError: 307548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8184.665717] Lustre: server umount lustre-MDT0001 complete [ 8188.393795] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8189.351028] Key type lgssc unregistered [ 8189.473319] LNet: 311316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8189.475372] LNetError: 311316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8189.484247] LNet: Removed LNI 192.168.206.153@tcp [ 8189.760800] Key type .llcrypt unregistered [ 8189.761707] Key type ._llcrypt unregistered [ 8221.304637] Key type ._llcrypt registered [ 8221.306524] Key type .llcrypt registered [ 8221.337792] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8221.419379] Key type .llcrypt unregistered [ 8221.421031] Key type ._llcrypt unregistered [ 8246.081791] Key type ._llcrypt registered [ 8246.083074] Key type .llcrypt registered [ 8246.118026] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8246.201286] Key type .llcrypt unregistered [ 8246.202113] Key type ._llcrypt unregistered [ 8256.171764] Key type ._llcrypt registered [ 8256.172970] Key type .llcrypt registered [ 8256.205489] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8256.297480] Key type .llcrypt unregistered [ 8256.298826] Key type ._llcrypt unregistered [ 8266.279374] Key type ._llcrypt registered [ 8266.280187] Key type .llcrypt registered [ 8266.312330] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8266.401191] Key type .llcrypt unregistered [ 8266.402443] Key type ._llcrypt unregistered [ 8291.922646] Key type ._llcrypt registered [ 8291.924382] Key type .llcrypt registered [ 8291.966170] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8292.071436] Key type .llcrypt unregistered [ 8292.073188] Key type ._llcrypt unregistered [ 8303.173600] Key type ._llcrypt registered [ 8303.174901] Key type .llcrypt registered [ 8303.214511] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8303.329215] Key type .llcrypt unregistered [ 8303.330464] Key type ._llcrypt unregistered [ 8314.108334] Key type ._llcrypt registered [ 8314.109893] Key type .llcrypt registered [ 8314.145217] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8314.241290] Key type .llcrypt unregistered [ 8314.242293] Key type ._llcrypt unregistered [ 8325.288288] Key type ._llcrypt registered [ 8325.289360] Key type .llcrypt registered [ 8325.327438] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8325.441169] Key type .llcrypt unregistered [ 8325.442280] Key type ._llcrypt unregistered [ 8336.468349] Key type ._llcrypt registered [ 8336.469370] Key type .llcrypt registered [ 8336.505907] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8336.609262] Key type .llcrypt unregistered [ 8336.611640] Key type ._llcrypt unregistered [ 8347.239087] Key type ._llcrypt registered [ 8347.239908] Key type .llcrypt registered [ 8347.271710] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8347.365253] Key type .llcrypt unregistered [ 8347.366243] Key type ._llcrypt unregistered [ 8366.787542] Key type ._llcrypt registered [ 8366.788412] Key type .llcrypt registered [ 8366.820643] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8366.917324] Key type .llcrypt unregistered [ 8366.918707] Key type ._llcrypt unregistered [ 8376.944275] Key type ._llcrypt registered [ 8376.945268] Key type .llcrypt registered [ 8376.976418] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8377.055382] Key type .llcrypt unregistered [ 8377.056659] Key type ._llcrypt unregistered [ 8386.818212] Key type ._llcrypt registered [ 8386.819205] Key type .llcrypt registered [ 8386.848912] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8386.927397] Key type .llcrypt unregistered [ 8386.928294] Key type ._llcrypt unregistered [ 8403.481072] Key type ._llcrypt registered [ 8403.482462] Key type .llcrypt registered [ 8403.515684] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 8409.431510] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 8409.993602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8410.016040] alg: No test for adler32 (adler32-zlib) [ 8410.867199] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8410.945557] LNet: Added LNI 192.168.206.153@tcp [8/256/0/180] [ 8412.519133] Key type lgssc registered [ 8412.848130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8423.450308] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8423.457552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.547688] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8424.559174] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8424.591569] Lustre: tst250fs-MDT0000: new disk, initializing [ 8424.614046] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8424.620471] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8425.856701] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8429.839818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.878454] Lustre: 330929:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8429.899025] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8429.901554] Lustre: Skipped 1 previous similar message [ 8429.932676] Lustre: tst250fs-MDT0001: new disk, initializing [ 8429.965166] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8429.978896] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8429.982371] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8431.273216] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8433.428846] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8435.415688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8435.503846] Lustre: tst250fs-OST0000: new disk, initializing [ 8435.505705] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8435.532663] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8437.133584] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8439.356024] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8439.359264] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8439.397326] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8441.116822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8441.186517] Lustre: tst250fs-OST0001: new disk, initializing [ 8441.190517] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8441.222683] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8443.154767] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug -1 all [ 8445.656392] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8449.138976] Lustre: 330931:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.206.153@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 8455.647636] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8455.649810] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.656066] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8456.672303] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8456.674528] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.679324] Lustre: Skipped 1 previous similar message [ 8461.788829] Lustre: server umount tst250fs-MDT0000 complete [ 8462.999567] LustreError: 330921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224273 with bad export cookie 58559869540152934 [ 8463.003298] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8463.005388] LustreError: 330921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8463.107373] Lustre: server umount tst250fs-MDT0001 complete [ 8475.144208] Lustre: server umount tst250fs-OST0000 complete [ 8486.684270] Lustre: server umount tst250fs-OST0001 complete [ 8492.694482] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 8495.410744] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 8505.472833] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 09:51:55 (1775224315) [ 8506.091359] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8511.372486] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid [ 8514.654148] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local [ 8524.948867] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8212 sec ========== 09:52:14 (1775224334) [ 8525.492790] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:52:15 (1775224335) === [ 8526.641745] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:52:16 (1775224336) === [ 8539.812288] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing unload_modules_local [ 8540.890821] Key type lgssc unregistered [ 8541.014323] LNet: 342374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8541.017245] LNetError: 342374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8541.025292] LNet: Removed LNI 192.168.206.153@tcp [ 8541.329102] Key type .llcrypt unregistered [ 8541.330170] Key type ._llcrypt unregistered