[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 657572456 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002540] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010095] LSM: Security Framework initializing [ 0.011065] Yama: becoming mindful. [ 0.012058] SELinux: Initializing. [ 0.014049] *** VALIDATE selinux *** [ 0.027259] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.036597] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.038016] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040030] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.041123] *** VALIDATE tmpfs *** [ 0.042720] *** VALIDATE proc *** [ 0.045068] *** VALIDATE cgroup *** [ 0.046014] *** VALIDATE cgroup2 *** [ 0.048296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.049194] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.050013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.051033] Spectre V2 : User space: Vulnerable [ 0.052016] Speculative Store Bypass: Vulnerable [ 0.057837] debug: unmapping init [mem 0xffffffff9dc59000-0xffffffff9dc60fff] [ 0.062000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.063894] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.064558] ... version: 2 [ 0.065017] ... bit width: 48 [ 0.066018] ... generic registers: 4 [ 0.067018] ... value mask: 0000ffffffffffff [ 0.068021] ... max period: 00007fffffffffff [ 0.069020] ... fixed-purpose events: 3 [ 0.070020] ... event mask: 000000070000000f [ 0.071339] rcu: Hierarchical SRCU implementation. [ 0.074519] smp: Bringing up secondary CPUs ... [ 0.076690] x86: Booting SMP configuration: [ 0.077025] .... node #0, CPUs: #1 #2 #3 [ 0.093571] smp: Brought up 1 node, 4 CPUs [ 0.095033] smpboot: Max logical packages: 1 [ 0.096017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.174860] node 0 deferred pages initialised in 75ms [ 0.186457] devtmpfs: initialized [ 0.191407] x86/mm: Memory block size: 128MB [ 0.194358] gcov: version magic: 0x41383552 [ 0.198328] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.199204] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.202081] pinctrl core: initialized pinctrl subsystem [ 0.206668] [ 0.208010] ************************************************************* [ 0.213015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220015] ** ** [ 0.229016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.234012] ** ** [ 0.240012] ** This means that this kernel is built to expose internal ** [ 0.244015] ** IOMMU data structures, which may compromise security on ** [ 0.248013] ** your system. ** [ 0.252011] ** ** [ 0.255012] ** If you see this message and you are not debugging the ** [ 0.259016] ** kernel, report this immediately to your vendor! ** [ 0.263022] ** ** [ 0.267015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270016] ************************************************************* [ 0.275295] NET: Registered protocol family 16 [ 0.278671] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.292061] cpuidle: using governor menu [ 0.295331] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299647] PCI: Using configuration type 1 for base access [ 0.303127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.316106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.318018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.320041] cryptd: max_cpu_qlen set to 1000 [ 0.323282] ACPI: Added _OSI(Module Device) [ 0.326020] ACPI: Added _OSI(Processor Device) [ 0.328015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.331019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.339432] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.351466] ACPI: Interpreter enabled [ 0.354069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.356015] ACPI: Using IOAPIC for interrupt routing [ 0.359174] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.364682] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.382444] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.385041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.389020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.393135] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.400837] acpiphp: Slot [2] registered [ 0.403154] acpiphp: Slot [5] registered [ 0.405133] acpiphp: Slot [6] registered [ 0.407263] acpiphp: Slot [7] registered [ 0.409174] acpiphp: Slot [8] registered [ 0.411116] acpiphp: Slot [9] registered [ 0.413161] acpiphp: Slot [10] registered [ 0.415158] acpiphp: Slot [3] registered [ 0.417129] acpiphp: Slot [4] registered [ 0.419135] acpiphp: Slot [11] registered [ 0.421120] acpiphp: Slot [12] registered [ 0.423318] acpiphp: Slot [13] registered [ 0.426249] acpiphp: Slot [14] registered [ 0.428134] acpiphp: Slot [15] registered [ 0.429106] acpiphp: Slot [16] registered [ 0.431205] acpiphp: Slot [17] registered [ 0.433143] acpiphp: Slot [18] registered [ 0.435358] acpiphp: Slot [19] registered [ 0.438114] acpiphp: Slot [20] registered [ 0.440153] acpiphp: Slot [21] registered [ 0.442307] acpiphp: Slot [22] registered [ 0.444120] acpiphp: Slot [23] registered [ 0.446316] acpiphp: Slot [24] registered [ 0.449237] acpiphp: Slot [25] registered [ 0.450283] acpiphp: Slot [26] registered [ 0.453115] acpiphp: Slot [27] registered [ 0.455217] acpiphp: Slot [28] registered [ 0.457115] acpiphp: Slot [29] registered [ 0.459131] acpiphp: Slot [30] registered [ 0.462131] acpiphp: Slot [31] registered [ 0.464065] PCI host bridge to bus 0000:00 [ 0.467027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.468000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.472097] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.477026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.481145] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.487045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.491195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.497072] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.504351] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.524019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.532068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.539083] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.543018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.546023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.549935] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.556209] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.560057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.565177] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.571016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.591016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.600023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.608183] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.620016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.637024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.673019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.693270] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.718023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.735019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.771015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.790251] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.816217] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.838021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.880033] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.903000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.913015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.929017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.967024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.981000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.993016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.003349] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.039024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.070961] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.101039] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.121030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.145031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.172855] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.188156] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.194000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.194468] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.197514] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.206101] iommu: Default domain type: Passthrough [ 1.210014] SCSI subsystem initialized [ 1.212894] ACPI: bus type USB registered [ 1.216144] usbcore: registered new interface driver usbfs [ 1.220125] usbcore: registered new interface driver hub [ 1.223103] usbcore: registered new device driver usb [ 1.229247] pps_core: LinuxPPS API ver. 1 registered [ 1.232023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.240135] PTP clock support registered [ 1.244134] EDAC MC: Ver: 3.0.0 [ 1.245459] PCI: Using ACPI for IRQ routing [ 1.246542] NetLabel: Initializing [ 1.249015] NetLabel: domain hash size = 128 [ 1.252015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.255093] NetLabel: unlabeled traffic allowed by default [ 1.257449] vgaarb: loaded [ 1.259971] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.261012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.268145] clocksource: Switched to clocksource kvm-clock [ 1.492278] VFS: Disk quotas dquot_6.6.0 [ 1.493864] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.496195] *** VALIDATE ramfs *** [ 1.497417] *** VALIDATE hugetlbfs *** [ 1.499092] pnp: PnP ACPI init [ 1.501502] pnp: PnP ACPI: found 6 devices [ 1.519205] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.522165] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.524239] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.526407] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.529156] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.531882] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.535299] NET: Registered protocol family 2 [ 1.538247] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.544453] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.548246] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.553334] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.558211] TCP: Hash tables configured (established 65536 bind 65536) [ 1.563973] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.570621] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.576476] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.583967] NET: Registered protocol family 1 [ 1.591984] RPC: Registered named UNIX socket transport module. [ 1.596305] RPC: Registered udp transport module. [ 1.599526] RPC: Registered tcp transport module. [ 1.603488] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.608694] NET: Registered protocol family 44 [ 1.611206] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.614547] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.619418] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.623380] PCI: CLS 0 bytes, default 64 [ 1.627167] Unpacking initramfs... [ 4.643394] debug: unmapping init [mem 0xffff9aa5bcc54000-0xffff9aa5bffbffff] [ 4.649207] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.652477] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.657953] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 5.560584] Initialise system trusted keyrings [ 5.567177] Key type blacklist registered [ 5.572340] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.584940] zbud: loaded [ 5.588819] *** VALIDATE nfs *** [ 5.590374] *** VALIDATE nfs4 *** [ 5.592515] pstore: using deflate compression [ 5.597111] Platform Keyring initialized [ 5.889802] NET: Registered protocol family 38 [ 5.892627] Key type asymmetric registered [ 5.894611] Asymmetric key parser 'x509' registered [ 5.899393] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.907670] io scheduler mq-deadline registered [ 5.910491] io scheduler kyber registered [ 5.914440] io scheduler bfq registered [ 5.917655] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.921612] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.925578] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.929258] ACPI: Power Button [PWRF] [ 5.936325] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.945310] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.965755] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.983439] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.012928] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.056845] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.090596] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.096918] Non-volatile memory driver v1.3 [ 6.099151] Linux agpgart interface v0.103 [ 6.147246] virtio_blk virtio1: [vda] 149392 512-byte logical blocks (76.5 MB/72.9 MiB) [ 6.150996] vda: detected capacity change from 0 to 76488704 [ 6.173582] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.177403] vdb: detected capacity change from 0 to 1073741824 [ 6.211622] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.217927] vdc: detected capacity change from 0 to 2621440000 [ 6.245453] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.248910] vdd: detected capacity change from 0 to 2621440000 [ 6.271131] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.275394] vde: detected capacity change from 0 to 4294967296 [ 6.305400] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.309369] vdf: detected capacity change from 0 to 4294967296 [ 6.318276] libphy: Fixed MDIO Bus: probed [ 6.327384] usbcore: registered new interface driver usbserial_generic [ 6.330860] usbserial: USB Serial support registered for generic [ 6.334756] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.340801] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.342930] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.345519] mousedev: PS/2 mouse device common for all mice [ 6.348867] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.352684] rtc_cmos 00:05: RTC can wake from S4 [ 6.353772] rtc_cmos 00:05: registered as rtc0 [ 6.353831] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.353863] intel_pstate: CPU model not supported [ 6.361655] hid: raw HID events driver (C) Jiri Kosina [ 6.368381] usbcore: registered new interface driver usbhid [ 6.370433] usbhid: USB HID core driver [ 6.371988] drop_monitor: Initializing network drop monitor service [ 6.373505] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.382861] Initializing XFRM netlink socket [ 6.390455] NET: Registered protocol family 10 [ 6.393873] Segment Routing with IPv6 [ 6.395514] NET: Registered protocol family 17 [ 6.397320] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.404843] mpls_gso: MPLS GSO support [ 6.415456] RAS: Correctable Errors collector initialized. [ 6.417972] AVX version of gcm_enc/dec engaged. [ 6.423763] AES CTR mode by8 optimization enabled [ 6.519467] sched_clock: Marking stable (6519388832, 0)->(8522385115, -2002996283) [ 6.523801] registered taskstats version 1 [ 6.526438] Loading compiled-in X.509 certificates [ 6.528920] zswap: loaded using pool lzo/zbud [ 6.558795] Key type big_key registered [ 6.576512] Key type encrypted registered [ 6.578482] ima: No TPM chip found, activating TPM-bypass! [ 6.582462] ima: Allocated hash algorithm: sha1 [ 6.584972] ima: No architecture policies found [ 6.587854] evm: Initialising EVM extended attributes: [ 6.591447] evm: security.selinux [ 6.593873] evm: security.ima [ 6.595935] evm: security.capability [ 6.598767] evm: HMAC attrs: 0x1 [ 6.602366] rtc_cmos 00:05: setting system clock to 2026-08-10 22:30:34 UTC (1786401034) [ 6.614821] debug: unmapping init [mem 0xffffffff9ec03000-0xffffffff9edfffff] [ 6.618714] debug: unmapping init [mem 0xffffffff9d982000-0xffffffff9dc58fff] [ 6.627348] Write protecting the kernel read-only data: 28672k [ 6.633339] debug: unmapping init [mem 0xffffffff9c003000-0xffffffff9c1fffff] [ 6.637837] debug: unmapping init [mem 0xffffffff9c914000-0xffffffff9c9fffff] [ 6.712570] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.733623] systemd[1]: Detected virtualization kvm. [ 6.739569] systemd[1]: Detected architecture x86-64. [ 6.742487] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.777700] systemd[1]: No hostname configured. [ 6.786250] systemd[1]: Set hostname to . [ 6.793530] random: systemd: uninitialized urandom read (16 bytes read) [ 6.796578] systemd[1]: Initializing machine ID from random generator. [ 7.362671] random: systemd: uninitialized urandom read (16 bytes read) [ 7.372930] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 7.396871] random: systemd: uninitialized urandom read (16 bytes read) [ 7.408667] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.435976] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.334928] device-mapper: uevent: version 1.0.3 [ 9.342184] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.355787] virtio_net virtio0 ens2: renamed from eth0 [ 11.654252] random: fast init done [ 12.112897] scsi host0: ata_piix [ 12.140586] scsi host1: ata_piix [ 12.149352] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.164612] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.118624] random: crng init done [ 16.120863] random: 7 urandom warning(s) missed due to ratelimiting [ 19.166787] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.441377] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.275235] printk: systemd: 25 output lines suppressed due to ratelimiting [ 24.177477] SELinux: Disabled at runtime. [ 24.312256] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.342660] systemd[1]: Detected virtualization kvm. [ 24.350377] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.289149] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.297435] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.312430] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.321918] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.334662] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.364811] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.371922] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System...[ 26.938700] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 28.522363] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.409716] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.614081] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.287105] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.399883] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 34.493028] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running f[ 35.345216] NFS: Registering the id_resolver key type or Configur…-only root support (9s / no limit)[ 35.350635] Key type id_resolver registered [ 35.365313] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg428-server login: [ 92.184076] libcfs: loading out-of-tree module taints kernel. [ 92.225846] Key type ._llcrypt registered [ 92.227577] Key type .llcrypt registered [ 92.298407] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 110.181357] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 111.984173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.011070] alg: No test for adler32 (adler32-zlib) [ 113.419931] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 114.212391] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 115.911179] Key type lgssc registered [ 117.772195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.646355] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.321353] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 176.563314] hrtimer: interrupt took 5146808 ns [ 189.866819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 189.889542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.169734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.195453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.266889] Lustre: lustre-MDT0000: new disk, initializing [ 191.345693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.357274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.289278] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.258076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.434559] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 211.481326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 211.485158] Lustre: Skipped 1 previous similar message [ 211.594153] Lustre: lustre-MDT0001: new disk, initializing [ 211.719531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.793410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.817839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.495945] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.938253] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.935756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.289084] Lustre: lustre-OST0000: new disk, initializing [ 235.293912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.313936] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.385955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.721455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.735431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 241.855289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.751983] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.606430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.768953] Lustre: lustre-OST0001: new disk, initializing [ 256.775539] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 256.782870] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.840592] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.183270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.191417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.304554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.395974] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.400918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.945413] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.741858] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.432468] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing yml_node [ 296.865416] Lustre: DEBUG MARKER: Client: 2.17.54.108 [ 299.486385] Lustre: DEBUG MARKER: MDS: 2.17.54.108 [ 301.938685] Lustre: DEBUG MARKER: OSS: 2.17.54.108 [ 303.373824] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 18:35:29 EDT 2026 [ 318.073524] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 320.187889] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.915203] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.915899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.936477] Lustre: Skipped 2 previous similar messages [ 338.955674] Lustre: Skipped 3 previous similar messages [ 342.268489] Lustre: server umount lustre-MDT0000 complete [ 345.746193] LustreError: 7462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401373 with bad export cookie 3199081919163407855 [ 345.753304] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.757825] LustreError: 7462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 349.151947] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.152412] LustreError: 12105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.162324] Lustre: Skipped 1 previous similar message [ 349.172855] LustreError: 12105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 349.175752] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.181857] Lustre: Skipped 1 previous similar message [ 350.179113] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.182364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.202250] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 350.208074] Lustre: Skipped 1 previous similar message [ 352.203723] Lustre: server umount lustre-MDT0001 complete [ 355.995399] Lustre: server umount lustre-OST0000 complete [ 359.655640] Lustre: server umount lustre-OST0001 complete [ 366.251972] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 376.582683] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 411.371179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.583889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.613280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.699905] Lustre: lustre-MDT0000: new disk, initializing [ 411.760923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.780893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.422808] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 427.618391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.819414] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 427.894613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 427.901449] Lustre: Skipped 1 previous similar message [ 427.987708] Lustre: lustre-MDT0001: new disk, initializing [ 428.149555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.199481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 428.218695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 433.848633] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 444.432228] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 451.294382] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 458.470252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 458.736392] Lustre: lustre-OST0000: new disk, initializing [ 458.741483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 458.747462] Lustre: 17046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 458.957417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.032583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 460.055633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 460.179919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 465.987449] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 476.024315] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 481.509345] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 481.911926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.381405] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.641635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 490.979893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.985639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.994254] Lustre: Skipped 2 previous similar messages [ 490.994574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 491.017287] Lustre: Skipped 1 previous similar message [ 494.729613] Lustre: server umount lustre-OST0000 complete [ 499.680520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.685625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.697896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.726327] Lustre: server umount lustre-MDT0000 complete [ 506.339509] LustreError: 15088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.359834] LustreError: 15088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 508.450747] LustreError: 15873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401536 with bad export cookie 3199081919163409381 [ 508.459276] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.470305] LustreError: 15873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 508.816994] Lustre: server umount lustre-MDT0001 complete [ 514.844903] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:39:00 (1786401540) [ 534.030567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.365802] LustreError: 19923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.451807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.466864] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 539.624211] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.997454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.800987] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 560.659290] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 566.161653] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.174321] Lustre: Failing over lustre-MDT0000 [ 568.385736] Lustre: server umount lustre-MDT0000 complete [ 568.803918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 568.811372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.820984] Lustre: Skipped 1 previous similar message [ 568.828147] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.846987] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 585.183382] Lustre: 3650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401597/real 1786401597] req@ffff9aa508155f80 x1873177364032896/t0(0) o400->MGC192.168.204.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786401613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 585.219855] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 585.245078] LustreError: 21158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.260419] LustreError: 21158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 589.095143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.427471] LustreError: 3646:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9aa635766a00 x1873177364035712/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 600.685401] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 601.061045] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 601.073983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 601.100501] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 610.541293] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 615.393779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.395685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 615.412068] Lustre: Skipped 2 previous similar messages [ 615.423439] Lustre: Skipped 2 previous similar messages [ 629.215162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 629.362763] Lustre: server umount lustre-MDT0000 complete [ 633.308971] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401661 with bad export cookie 3199081919163410802 [ 633.323698] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.332330] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 633.672269] Lustre: server umount lustre-MDT0001 complete [ 642.033847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.613665] LustreError: 23265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.661310] LustreError: 23265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 642.787867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.794196] Lustre: Skipped 2 previous similar messages [ 648.033326] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 657.238741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.382133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 657.389674] Lustre: Skipped 6 previous similar messages [ 662.646456] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 671.445219] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.104357] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 680.182973] Lustre: Failing over lustre-MDT0000 [ 680.192054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.361219] Lustre: server umount lustre-MDT0000 complete [ 698.860376] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401710/real 1786401710] req@ffff9aa6358cea00 x1873177364063360/t0(0) o400->MGC192.168.204.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786401726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 698.931346] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.828377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.193463] LustreError: 3646:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9aa6358cea00 x1873177364065792/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 708.222050] LustreError: 24424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.258718] LustreError: 24424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 708.697975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 713.703371] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 713.731071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 713.751753] Lustre: Skipped 1 previous similar message [ 713.797472] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 714.248743] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 726.031864] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 733.157807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.168349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.183091] Lustre: Skipped 2 previous similar messages [ 733.217679] Lustre: Skipped 3 previous similar messages [ 736.755534] Lustre: server umount lustre-MDT0000 complete [ 741.449311] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401769 with bad export cookie 3199081919163411740 [ 741.467996] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.915152] Lustre: server umount lustre-MDT0001 complete [ 753.400408] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:42:59 (1786401779) [ 770.237982] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 781.724381] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 826.283821] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 840.257410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.542909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 840.571979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 840.642682] Lustre: lustre-MDT0000: new disk, initializing [ 840.721412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.735818] Lustre: Skipped 2 previous similar messages [ 840.757273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 845.565439] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 857.728351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.821770] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 858.100448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 863.590704] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 874.677823] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 880.721741] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 888.473658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 888.742165] Lustre: 31612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 890.862625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 890.989949] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 891.195533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 891.244664] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 896.854977] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 907.714628] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 913.255722] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 932.324550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.335042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.357168] Lustre: Skipped 1 previous similar message [ 932.376587] Lustre: Skipped 1 previous similar message [ 937.885464] Lustre: server umount lustre-OST0000 complete [ 945.120374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 949.408592] Lustre: server umount lustre-MDT0000 complete [ 952.817856] LustreError: 29649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.853411] LustreError: 29649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 954.230709] LustreError: 32846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401982 with bad export cookie 3199081919163412146 [ 954.240611] LustreError: 32846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 954.248766] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.707576] Lustre: server umount lustre-MDT0001 complete [ 969.472680] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 973.789743] Key type lgssc unregistered [ 974.307156] LNet: 33597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.316181] LNetError: 33597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.335924] LNet: Removed LNI 192.168.204.128@tcp [ 976.718375] Key type .llcrypt unregistered [ 976.724319] Key type ._llcrypt unregistered [ 1008.999851] Key type ._llcrypt registered [ 1009.001622] Key type .llcrypt registered [ 1009.140614] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1010.680420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1010.921637] alg: No test for adler32 (adler32-zlib) [ 1012.515841] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1012.899528] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1014.719202] Key type lgssc registered [ 1016.660882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.759713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.803365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.222876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.123605] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1037.796432] LustreError: 35012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.817763] LustreError: 35012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1042.499889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.806112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1047.812959] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1056.433546] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1063.278329] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1070.741712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1071.255123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.032402] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1087.798324] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1087.969497] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1102.641701] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:48:48 (1786402128) [ 1104.195986] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1105.730327] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1111.230660] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:48:57 (1786402137) [ 1121.792336] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:49:07 (1786402147) [ 1122.863662] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1131.573488] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1131.579034] Lustre: Skipped 2 previous similar messages [ 1133.711532] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:49:19 (1786402159) [ 1135.837870] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1135.842565] Lustre: Skipped 2 previous similar messages [ 1144.462511] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1144.471376] Lustre: Skipped 4 previous similar messages [ 1160.025736] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:49:45 (1786402185) [ 1163.204454] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1167.292924] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1177.922303] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:50:03 (1786402203) [ 1180.419103] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1180.425581] Lustre: Skipped 6 previous similar messages [ 1194.196754] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:50:19 (1786402219) [ 1210.851000] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.855853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1210.882725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.898936] Lustre: Skipped 1 previous similar message [ 1213.433955] Lustre: server umount lustre-OST0000 complete [ 1221.088777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.091164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1221.105325] Lustre: Skipped 2 previous similar messages [ 1221.129423] Lustre: Skipped 1 previous similar message [ 1225.680034] Lustre: server umount lustre-MDT0000 complete [ 1226.212275] LustreError: 40117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.240772] LustreError: 40117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1229.416029] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402257 with bad export cookie 18044926563702892096 [ 1229.425664] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.428023] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1229.897281] Lustre: server umount lustre-MDT0001 complete [ 1242.754817] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1245.999641] Key type lgssc unregistered [ 1246.266291] LNet: 42873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.276637] LNetError: 42873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.290946] LNet: Removed LNI 192.168.204.128@tcp [ 1247.280351] Key type .llcrypt unregistered [ 1247.281975] Key type ._llcrypt unregistered [ 1267.357162] Key type ._llcrypt registered [ 1267.358772] Key type .llcrypt registered [ 1267.478325] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 1288.517266] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1290.287597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1290.316866] alg: No test for adler32 (adler32-zlib) [ 1291.558604] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1291.839150] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1293.575280] Key type lgssc registered [ 1294.723409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.839695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.872431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.294991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1334.361978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1334.467499] Lustre: lustre-MDT0000: new disk, initializing [ 1334.537430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.555685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1339.900114] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1349.578815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.799783] Lustre: 46029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1349.868794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1349.873728] Lustre: Skipped 1 previous similar message [ 1349.988393] Lustre: lustre-MDT0001: new disk, initializing [ 1350.112442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.170094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1350.189432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1355.864593] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1365.734440] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1371.415453] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1378.996041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1379.226166] Lustre: lustre-OST0000: new disk, initializing [ 1379.229131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1379.233497] Lustre: 47998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1379.315218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.679990] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1388.121652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1388.131937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1388.267911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1397.969141] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1405.065592] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1405.356886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1410.501759] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1410.801961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1414.120279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1414.127793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.149641] LustreError: Skipped 1 previous similar message [ 1414.159030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1414.166403] Lustre: Skipped 1 previous similar message [ 1419.160467] Lustre: server umount lustre-OST0000 complete [ 1424.353609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.367388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.373268] Lustre: Skipped 2 previous similar messages [ 1429.472889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.475452] Lustre: Skipped 1 previous similar message [ 1434.595628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.601292] Lustre: Skipped 1 previous similar message [ 1438.175333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1438.384581] Lustre: server umount lustre-MDT0000 complete [ 1442.145738] LustreError: 46022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402470 with bad export cookie 1222295756818065042 [ 1442.147788] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.155806] LustreError: 46022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1442.615954] Lustre: server umount lustre-MDT0001 complete [ 1452.633657] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:54:38 (1786402478) [ 1459.944050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.452652] LustreError: 50462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.477529] LustreError: 50462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1460.590561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.828617] LustreError: 50463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.114825] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1470.946753] LustreError: 50462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.464947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.784772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1481.491843] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1489.672672] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1497.780868] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1505.525586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1505.804162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.029842] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1522.481151] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1527.631457] LustreError: 52991:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1528.443911] Lustre: Modifying parameter general.timeout=1 in log params [ 1529.548236] Lustre: Modifying parameter general.timeout=2 in log params [ 1531.845958] Lustre: Modifying parameter general.timeout=4 in log params [ 1531.854109] Lustre: Skipped 1 previous similar message [ 1535.961946] Lustre: Modifying parameter general.timeout=8 in log params [ 1535.965951] Lustre: Skipped 3 previous similar messages [ 1545.244353] Lustre: Modifying parameter general.timeout=17 in log params [ 1545.250221] Lustre: Skipped 8 previous similar messages [ 1562.117400] Lustre: Modifying parameter general.timeout=34 in log params [ 1562.121208] Lustre: Skipped 16 previous similar messages [ 1594.363666] Lustre: Modifying parameter general.timeout=64 in log params [ 1594.368712] Lustre: Skipped 29 previous similar messages [ 1658.449636] Lustre: Modifying parameter general.timeout=125 in log params [ 1658.456188] Lustre: Skipped 60 previous similar messages [ 1676.851123] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:58:22 (1786402702) [ 1680.138250] Lustre: 59947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1682.403494] Lustre: 60047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1684.666577] Lustre: 60145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1690.215618] Lustre: 60356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1690.240564] Lustre: 60356:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1724.522223] Lustre: 61825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1724.541587] Lustre: 61825:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1724.581940] LustreError: 61825:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1729.295150] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:59:14 (1786402754) [ 1741.797537] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.801617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1741.823639] Lustre: Skipped 2 previous similar messages [ 1741.838544] Lustre: Skipped 2 previous similar messages [ 1747.265350] Lustre: server umount lustre-MDT0000 complete [ 1750.983397] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402778 with bad export cookie 1222295756818065924 [ 1750.990131] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1750.994951] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1751.346856] Lustre: server umount lustre-MDT0001 complete [ 1765.966150] Lustre: server umount lustre-OST0000 complete [ 1796.742138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.055200] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1797.087452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1797.096120] Lustre: Skipped 27 previous similar messages [ 1797.376268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.599057] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1813.468810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.539834] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1813.565322] Lustre: 64356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1819.411927] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1827.904716] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1835.157889] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1844.386597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1844.542468] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1844.954718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.963111] Lustre: Skipped 1 previous similar message [ 1852.748677] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1861.319756] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1863.447450] Lustre: 66939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1875.443385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.473214] Lustre: Skipped 1 previous similar message [ 1875.488724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.506041] Lustre: Skipped 3 previous similar messages [ 1878.853919] Lustre: server umount lustre-OST0000 complete [ 1885.667910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1885.676045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.696804] Lustre: Skipped 2 previous similar messages [ 1890.147582] Lustre: server umount lustre-MDT0000 complete [ 1890.787136] LustreError: 65104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.821614] LustreError: 65104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1895.667687] LustreError: 64350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402923 with bad export cookie 1222295756818069130 [ 1895.672937] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1895.696502] LustreError: 64350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1896.292583] Lustre: server umount lustre-MDT0001 complete [ 1910.624793] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1914.815922] Key type lgssc unregistered [ 1915.195790] LNet: 68147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.204210] LNetError: 68147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.222656] LNet: Removed LNI 192.168.204.128@tcp [ 1916.547280] Key type .llcrypt unregistered [ 1916.549362] Key type ._llcrypt unregistered [ 1954.828310] Key type ._llcrypt registered [ 1954.835366] Key type .llcrypt registered [ 1955.139684] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1957.921021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1957.986843] alg: No test for adler32 (adler32-zlib) [ 1959.224350] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1959.622275] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1961.359267] Key type lgssc registered [ 1963.662966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1974.486419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1974.502328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.189796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.327379] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1985.512476] LustreError: 69567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.538955] LustreError: 69567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1990.636401] LustreError: 69568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.639748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.748799] LustreError: 69567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.836481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2003.722640] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2014.075648] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2021.705331] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2031.258503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2032.075330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.555947] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2049.632773] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2058.882686] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2071.012072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.028774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.057561] Lustre: Skipped 1 previous similar message [ 2075.701618] Lustre: server umount lustre-MDT0000 complete [ 2076.144071] LustreError: 70919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.192412] LustreError: 70919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2081.253205] LustreError: 69568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.292632] LustreError: 69568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2081.689368] LustreError: 69548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403109 with bad export cookie 159726933034450173 [ 2081.718288] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2082.060058] Lustre: server umount lustre-MDT0001 complete [ 2098.190489] Lustre: server umount lustre-OST0000 complete [ 2136.200939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.689463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2136.762780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2137.151775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.451085] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2157.374571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.447127] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2157.467255] Lustre: 74442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2157.981924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.599120] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2178.082837] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2184.563315] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2194.948094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2195.399602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2196.041353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.692972] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2215.348081] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2217.634602] Lustre: 77031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2220.536226] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2230.246168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.249185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2230.264249] Lustre: Skipped 1 previous similar message [ 2230.272312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.276366] LustreError: Skipped 1 previous similar message [ 2230.291603] Lustre: Skipped 3 previous similar messages [ 2233.585744] Lustre: server umount lustre-OST0000 complete [ 2239.969671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2239.980063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.999628] Lustre: Skipped 1 previous similar message [ 2240.010441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.602702] Lustre: server umount lustre-MDT0000 complete [ 2245.603413] LustreError: 75403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.639733] LustreError: 75403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2249.317290] LustreError: 74435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403277 with bad export cookie 159726933034451013 [ 2249.325876] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2249.328928] LustreError: 74435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2249.665677] Lustre: server umount lustre-MDT0001 complete [ 2264.820961] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2269.147992] Key type lgssc unregistered [ 2269.806647] LNet: 78247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.820324] LNetError: 78247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.842849] LNet: Removed LNI 192.168.204.128@tcp [ 2271.342134] Key type .llcrypt unregistered [ 2271.345377] Key type ._llcrypt unregistered [ 2311.581517] Key type ._llcrypt registered [ 2311.587682] Key type .llcrypt registered [ 2311.789369] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2314.176610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2314.609224] alg: No test for adler32 (adler32-zlib) [ 2316.574905] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 2316.944620] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2318.799284] Key type lgssc registered [ 2321.079256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2333.420341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2333.458049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.929359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.781091] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2344.425031] LustreError: 79666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.458496] LustreError: 79666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2349.537969] LustreError: 79667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.213886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.062664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2359.329122] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2369.587206] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2376.277881] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2384.365941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.030118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.054913] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2404.741244] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2411.918605] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2413.167567] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2414.594586] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2416.984795] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2416.989131] Lustre: Skipped 1 previous similar message [ 2421.071798] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2421.079979] Lustre: Skipped 2 previous similar messages [ 2429.828660] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2429.832654] Lustre: Skipped 6 previous similar messages [ 2446.545776] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2446.549952] Lustre: Skipped 14 previous similar messages [ 2479.226686] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2479.230663] Lustre: Skipped 22 previous similar messages [ 2548.589875] Lustre: 87790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2549.838842] Lustre: Disabling parameter general.jobid_var= in log params [ 2549.842890] Lustre: Skipped 47 previous similar messages [ 2552.430670] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:12:57 (1786403577) [ 2554.610538] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2557.559454] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:13:02 (1786403582) [ 2563.557446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.565333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2563.591434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.604714] Lustre: Skipped 1 previous similar message [ 2568.680623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.685901] Lustre: Skipped 1 previous similar message [ 2573.794063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.803312] Lustre: Skipped 1 previous similar message [ 2576.351665] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2576.555896] Lustre: server umount lustre-OST0000 complete [ 2583.011255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2583.022615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.039312] Lustre: Skipped 1 previous similar message [ 2583.048396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.041756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.162484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2589.167498] Lustre: Skipped 2 previous similar messages [ 2596.319138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2596.460746] Lustre: server umount lustre-MDT0000 complete [ 2599.393785] LustreError: 79666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.404106] LustreError: 79666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2600.603263] LustreError: 79648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403628 with bad export cookie 15275015270569305075 [ 2600.604732] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.612396] LustreError: 79648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2600.859474] Lustre: server umount lustre-MDT0001 complete [ 2615.150878] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2619.558286] Key type lgssc unregistered [ 2619.957563] LNet: 89042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.985623] LNetError: 89042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.030327] LNet: Removed LNI 192.168.204.128@tcp [ 2622.190793] Key type .llcrypt unregistered [ 2622.193734] Key type ._llcrypt unregistered [ 2630.323640] Key type ._llcrypt registered [ 2630.326548] Key type .llcrypt registered [ 2630.435611] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2639.250625] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules [ 2641.474991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2641.526158] alg: No test for adler32 (adler32-zlib) [ 2642.564656] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 2642.855868] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2644.567182] Key type lgssc registered [ 2647.929980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2650.883756] LustreError: 90342:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2672.218360] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2674.791191] LustreError: 90342:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2685.138865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2685.166940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.575313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.486438] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2696.163557] LustreError: 91369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.197576] LustreError: 91369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2701.284175] LustreError: 91368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.661441] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:15:28 (1786403728) [ 2706.406345] LustreError: 91369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.750459] LustreError: 91400:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2707.758793] LustreError: 91400:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2707.767909] LustreError: 91400:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2708.069377] Lustre: server umount lustre-MDT0000 complete [ 2724.267782] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2727.366343] Key type lgssc unregistered [ 2727.602630] LNet: 92894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2727.610500] LNetError: 92894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2727.625905] LNet: Removed LNI 192.168.204.128@tcp [ 2728.488980] Key type .llcrypt unregistered [ 2728.491475] Key type ._llcrypt unregistered [ 2749.590140] Key type ._llcrypt registered [ 2749.593493] Key type .llcrypt registered [ 2749.734248] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2751.349271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2751.466779] alg: No test for adler32 (adler32-zlib) [ 2752.975635] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 2753.340465] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2755.199220] Key type lgssc registered [ 2756.414723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2764.899724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2764.915251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.613987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.258179] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2777.063075] LustreError: 94111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.102405] LustreError: 94111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2780.797180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.636656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.128598] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2796.426410] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.954441] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2812.122130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2812.778925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.090685] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2828.295346] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2885.127246] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:18:30 (1786403910) [ 2887.224400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2889.409660] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:18:34 (1786403914) [ 2897.894436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.912810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2897.915879] Lustre: Skipped 2 previous similar messages [ 2897.934646] Lustre: Skipped 2 previous similar messages [ 2900.245713] Lustre: server umount lustre-MDT0000 complete [ 2903.013167] LustreError: 94111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.034889] LustreError: 94111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2904.399518] LustreError: 94091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403932 with bad export cookie 14139624758939071966 [ 2904.409856] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2904.430811] LustreError: 94091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2904.834186] Lustre: server umount lustre-MDT0001 complete [ 2919.425231] Lustre: server umount lustre-OST0000 complete [ 2932.441548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.873217] LustreError: 98405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.050952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.343824] LustreError: 98406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.371255] LustreError: 98406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2938.583603] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2942.439728] LustreError: 98405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.350659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.938687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2954.694995] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2962.861129] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2970.033584] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2986.165240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.429641] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2986.434691] LustreError: 98394:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2986.478105] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2986.487946] LustreError: 100493:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2986.498399] LustreError: 100493:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2986.504643] LustreError: 100493:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2986.622637] Lustre: server umount lustre-OST0000 complete [ 2986.624305] LustreError: 100493:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2996.960658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.218023] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2997.232964] LustreError: 98393:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2997.265637] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2997.281805] LustreError: 100961:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2997.294992] LustreError: 100961:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2997.307589] LustreError: 100961:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2997.437507] Lustre: server umount lustre-OST0000 complete [ 2997.443651] LustreError: 100961:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3008.792669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.925982] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3008.934766] Lustre: Found index 0 for lustre-OST0000, updating log [ 3008.940263] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3009.001518] Lustre: lustre-OST0000: new disk, initializing [ 3009.003899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3009.008616] Lustre: 101497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3009.309589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.118534] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3019.984724] Lustre: Failing over lustre-OST0000 [ 3020.161324] Lustre: server umount lustre-OST0000 complete [ 3025.379031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3025.388987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.408423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.961051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.969688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.983868] Lustre: Skipped 1 previous similar message [ 3030.550924] Lustre: server umount lustre-MDT0000 complete [ 3034.082581] LustreError: 98406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3034.100088] LustreError: 98406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3034.837778] LustreError: 100513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404062 with bad export cookie 14139624758939074710 [ 3034.849262] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3034.857187] LustreError: 100513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3044.467701] Lustre: server umount lustre-MDT0001 complete [ 3055.025241] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:21:20 (1786404080) [ 3075.474813] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3088.849618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.462370] LustreError: 103843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.708560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.214705] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3102.956620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.162542] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3112.285312] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3119.121829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3119.802911] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3125.681306] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3128.940417] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3128.980934] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3128.984347] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3134.795685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.892092] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3134.975521] Lustre: lustre-OST0001: new disk, initializing [ 3134.979365] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3134.984990] Lustre: 106161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3135.280153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3135.287039] Lustre: Skipped 2 previous similar messages [ 3141.172072] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3142.272441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3142.279704] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3142.479521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3149.742178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3160.889537] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3165.155102] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3165.163594] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.188433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3169.401633] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [ 3169.407412] Lustre: Skipped 3 previous similar messages [ 3171.003978] Lustre: server umount lustre-MDT0001 complete [ 3173.350857] LustreError: 105100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.384583] LustreError: 105100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3181.030584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.127275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3181.130921] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3181.170910] Lustre: 103831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3182.901864] LustreError: 107710:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3182.913036] LustreError: 107710:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3182.929469] LustreError: 103836:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3183.064645] LustreError: 107720:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -17 [ 3183.076749] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3183.076749] [ 3186.568111] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3187.184822] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3198.553370] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:23:43 (1786404223) [ 3218.924197] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3236.738441] Lustre: 109779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3259.142530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3262.746616] Lustre: 110914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3350.495874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3350.506869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.520984] Lustre: Skipped 2 previous similar messages [ 3350.526770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3355.631697] Lustre: server umount lustre-MDT0000 complete [ 3356.133032] LustreError: 103839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.155039] LustreError: 103839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3359.495116] LustreError: 106160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404387 with bad export cookie 14139624758939075711 [ 3359.499644] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3359.520906] LustreError: 106160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3360.046143] Lustre: server umount lustre-MDT0001 complete [ 3374.268531] Lustre: server umount lustre-OST0000 complete [ 3377.183233] Lustre: 93421:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404389/real 1786404389] req@ffff9aa510c3ea00 x1873180131432704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786404405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3377.203055] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.213029] Lustre: Skipped 3 previous similar messages [ 3377.999063] Lustre: server umount lustre-OST0001 complete [ 3383.678450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3394.252895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3403.852875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3415.791265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3441.851513] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3453.699706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.735143] Lustre: lustre-MDT0000: reset Object Index mappings [ 3454.289460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.296562] Lustre: Skipped 1 previous similar message [ 3458.681470] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3467.380881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.398581] Lustre: lustre-MDT0001: reset Object Index mappings [ 3473.106498] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3476.081731] Lustre: 116022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3482.871551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.115637] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3493.122133] LustreError: 116377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.166573] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3493.174284] LustreError: 116377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3498.327073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3499.449839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.147018] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3505.156015] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3505.836198] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3513.563832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3517.383635] Lustre: 117897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3572.573973] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:29:58 (1786404598) [ 3600.868824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3600.880847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.905059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3600.913932] Lustre: Skipped 4 previous similar messages [ 3605.074503] Lustre: server umount lustre-MDT0000 complete [ 3609.032576] LustreError: 114854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404636 with bad export cookie 14139624758939122583 [ 3609.051632] LustreError: 114854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3609.053684] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3615.649024] Lustre: server umount lustre-MDT0001 complete [ 3626.880235] Lustre: server umount lustre-OST0000 complete [ 3630.792388] LustreError: 118855:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x3c:0x0].0x0 (ffff9aa608088400) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3645.407487] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3645.652805] Lustre: server umount lustre-OST0001 complete [ 3653.824639] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3666.237316] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3708.751612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.974472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3708.999436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3709.068619] Lustre: lustre-MDT0000: new disk, initializing [ 3709.132624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.138257] Lustre: Skipped 3 previous similar messages [ 3709.151555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3714.480844] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3721.958594] Lustre: server umount lustre-MDT0000 complete [ 3734.348686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.704421] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3734.721756] Lustre: 122168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3734.741723] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3740.519817] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3746.666584] Lustre: server umount lustre-MDT0000 complete [ 3759.784983] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:33:04 (1786404784) [ 3762.435877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3764.856619] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:33:10 (1786404790) [ 3790.868750] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3804.092711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.465436] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3818.405380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3818.560695] Lustre: 124243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3818.616833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3818.625305] Lustre: Skipped 1 previous similar message [ 3818.733214] Lustre: lustre-MDT0001: new disk, initializing [ 3818.905862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3818.914828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3824.870952] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3829.972497] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3837.282151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3837.605921] Lustre: lustre-OST0000: new disk, initializing [ 3837.610732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3837.621514] Lustre: 125821:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3839.683144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3839.705596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3839.864843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3845.526314] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3857.064899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3857.229520] Lustre: lustre-OST0001: new disk, initializing [ 3857.233585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3857.243234] Lustre: 126692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3858.559369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3858.579058] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3858.778312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3864.886223] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3875.230840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3879.222796] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4308.343531] Lustre: 133118:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4308.349447] Lustre: 133118:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4308.354218] Lustre: 133118:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4308.359502] Lustre: 133118:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4308.364840] Lustre: 133118:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4308.369760] Lustre: 133118:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4595.138272] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:46:59 (1786405619) [ 4613.092610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4613.103398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.120334] Lustre: Skipped 5 previous similar messages [ 4613.132087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.141871] Lustre: Skipped 8 previous similar messages [ 4617.715175] Lustre: server umount lustre-MDT0000 complete [ 4619.754188] LustreError: 127464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4619.789486] LustreError: 127464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 4622.165719] LustreError: 124234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405650 with bad export cookie 14139624758939191638 [ 4622.170326] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.174721] LustreError: 124234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4624.869779] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4624.872636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4624.890266] Lustre: Skipped 4 previous similar messages [ 4624.908229] Lustre: Skipped 5 previous similar messages [ 4628.627870] Lustre: server umount lustre-MDT0001 complete [ 4640.606800] Lustre: server umount lustre-OST0000 complete [ 4660.191133] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4660.568587] Lustre: server umount lustre-OST0001 complete [ 4668.217916] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4682.159934] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4730.729338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.158186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4731.237992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4731.359750] Lustre: lustre-MDT0000: new disk, initializing [ 4731.483550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.493622] Lustre: Skipped 5 previous similar messages [ 4731.512071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4737.295473] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4749.811221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.987440] Lustre: 136176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4750.075104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4750.081176] Lustre: Skipped 1 previous similar message [ 4750.157682] Lustre: lustre-MDT0001: new disk, initializing [ 4750.276422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4750.281882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4754.973357] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4765.457779] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4771.415491] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4780.172437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.574801] Lustre: lustre-OST0000: new disk, initializing [ 4780.583530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4780.596800] Lustre: 138145:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4780.697763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.702112] Lustre: Skipped 1 previous similar message [ 4782.049504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4782.077437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4782.223753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4789.149460] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4800.679552] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.835562] Lustre: lustre-MDD0000: changelog on [ 4814.184508] Lustre: lustre-MDD0001: changelog on [ 6808.101937] Lustre: lustre-MDD0001: changelog off [ 6810.536459] Lustre: 139602:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6811.410412] Lustre: lustre-MDD0000: changelog off [ 6819.817683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6819.820442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.828839] LustreError: Skipped 1 previous similar message [ 6819.867368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6819.878502] Lustre: Skipped 3 previous similar messages [ 6823.528517] Lustre: server umount lustre-OST0000 complete [ 6829.027026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6829.034925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.047941] Lustre: Skipped 1 previous similar message [ 6829.052608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6833.847853] Lustre: server umount lustre-MDT0000 complete [ 6835.170183] LustreError: 136182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.203920] LustreError: 136182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6837.871800] LustreError: 138144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407865 with bad export cookie 14139624758939978606 [ 6837.880150] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6837.896769] LustreError: 138144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6838.222246] Lustre: server umount lustre-MDT0001 complete [ 6849.445944] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6852.876773] Key type lgssc unregistered [ 6853.171549] LNet: 147301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6853.179959] LNetError: 147301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6853.201049] LNet: Removed LNI 192.168.204.128@tcp [ 6854.224227] Key type .llcrypt unregistered [ 6854.234321] Key type ._llcrypt unregistered [ 6876.780168] Key type ._llcrypt registered [ 6876.782218] Key type .llcrypt registered [ 6876.896357] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 6897.133138] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6899.987559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6900.047612] alg: No test for adler32 (adler32-zlib) [ 6901.111403] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 6901.395056] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6903.136135] Key type lgssc registered [ 6904.798694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6945.775761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6945.795263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.129424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6947.165380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6947.236789] Lustre: lustre-MDT0000: new disk, initializing [ 6947.343669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.357177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6951.515187] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6964.006934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.138698] Lustre: 150507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6964.198360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6964.202681] Lustre: Skipped 1 previous similar message [ 6964.278766] Lustre: lustre-MDT0001: new disk, initializing [ 6964.385862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6964.427236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6964.443206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6969.997625] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6980.110428] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6987.604837] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6996.975621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6997.195638] Lustre: lustre-OST0000: new disk, initializing [ 6997.199258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6997.205358] Lustre: 152477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6997.283381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.321253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7003.330207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7003.654713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7006.347984] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7020.171731] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7029.571726] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_module obdecho/obdecho [ 7032.095634] LustreError: 153481:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7032.102675] LustreError: 153481:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7034.077175] LustreError: 153577:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7051.746616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7051.755641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7051.772587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.771621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.786029] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.813947] Lustre: Skipped 1 previous similar message [ 7056.302493] Lustre: server umount lustre-MDT0000 complete [ 7061.004672] LustreError: 150499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408088 with bad export cookie 7170016517861034500 [ 7061.016651] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7061.022774] LustreError: 150499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7061.391993] Lustre: server umount lustre-MDT0001 complete [ 7077.153963] Lustre: server umount lustre-OST0000 complete [ 7105.831245] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 7118.289967] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7163.483883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.707774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7163.738792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7163.816418] Lustre: lustre-MDT0000: new disk, initializing [ 7163.897789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7163.923604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7169.092324] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7181.367462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.463539] Lustre: 157295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7181.546414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7181.550250] Lustre: Skipped 1 previous similar message [ 7181.604659] Lustre: lustre-MDT0001: new disk, initializing [ 7181.682341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7181.733950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7181.740627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7186.448639] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7197.771645] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7204.864836] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7213.440306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.844792] Lustre: lustre-OST0000: new disk, initializing [ 7213.850967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7213.864680] Lustre: 159267:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7214.070181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.857624] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7224.023894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7224.045412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7224.394348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7236.726290] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7244.222663] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7244.517738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7251.368408] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7251.599505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7255.015388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.017870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.033629] LustreError: Skipped 1 previous similar message [ 7255.038585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.075468] Lustre: Skipped 1 previous similar message [ 7255.132866] Lustre: Skipped 3 previous similar messages [ 7260.154377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.161233] Lustre: Skipped 1 previous similar message [ 7265.253479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7265.256039] Lustre: Skipped 1 previous similar message [ 7268.320376] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7268.556646] Lustre: server umount lustre-OST0000 complete [ 7273.954951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7273.963388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.981582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7287.264436] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7287.491563] Lustre: server umount lustre-MDT0000 complete [ 7290.849891] LustreError: 158521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7290.877538] LustreError: 158521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7291.414919] LustreError: 157288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408319 with bad export cookie 7170016517861035452 [ 7291.417145] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7291.421517] LustreError: 157288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7291.777391] Lustre: server umount lustre-MDT0001 complete [ 7295.498433] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:32:01 (1786408321) [ 7297.597579] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7300.416726] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:32:05 (1786408325) [ 7310.692931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.372271] LustreError: 161688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.441471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7316.449849] LustreError: 161689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7317.531929] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7321.571257] LustreError: 161688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.701174] LustreError: 161689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.282800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.671689] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7343.994357] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7351.274063] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7359.062640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.453820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.461715] Lustre: Skipped 1 previous similar message [ 7367.344995] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7375.897601] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7395.298428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7395.306600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.320035] Lustre: Skipped 1 previous similar message [ 7395.324945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7395.330080] Lustre: Skipped 6 previous similar messages [ 7399.149462] Lustre: server umount lustre-MDT0000 complete [ 7402.074910] LustreError: 164132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.262178] LustreError: 161670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408431 with bad export cookie 7170016517861036341 [ 7403.266864] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7403.269270] LustreError: 161670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7407.589261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.606203] Lustre: Skipped 2 previous similar messages [ 7409.892870] Lustre: server umount lustre-MDT0001 complete [ 7416.955102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7432.307218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.597711] LustreError: 165552:0:(llog.c:1655:llog_backup()) MGC192.168.204.128@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7432.607815] Lustre: 165552:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.128@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7433.184195] LustreError: 165557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7433.201970] LustreError: 165557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7468.634736] LustreError: 165559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7468.661795] LustreError: 165559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 7493.599309] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7493.606787] Lustre: 165552:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7533.143823] LustreError: 165559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.157601] LustreError: 165559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 7534.579359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7539.693352] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7539.815259] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7550.918303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.382745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7573.536483] LustreError: 148727:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000078b37cbb x1873184480713216/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7579.140968] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7579.381314] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7590.580242] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7593.313714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7601.750509] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7604.031820] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7625.191450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7625.204360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.209432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7625.223947] Lustre: Skipped 2 previous similar messages [ 7625.251750] Lustre: Skipped 9 previous similar messages [ 7626.703863] Lustre: server umount lustre-MDT0000 complete [ 7631.977821] LustreError: 161669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408659 with bad export cookie 7170016517861042830 [ 7631.990312] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7631.996195] LustreError: 161669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7632.382624] Lustre: server umount lustre-MDT0001 complete [ 7648.098404] Lustre: server umount lustre-OST0000 complete [ 7672.730615] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 7684.921361] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7727.856240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.135520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7728.169843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7728.234701] Lustre: lustre-MDT0000: new disk, initializing [ 7728.327418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.336817] Lustre: Skipped 1 previous similar message [ 7728.366081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7734.464931] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7747.643780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.753172] Lustre: 170666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7748.280665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7754.279516] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7765.922233] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7773.038392] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7782.025996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.355265] Lustre: lustre-OST0000: new disk, initializing [ 7782.357495] Lustre: Skipped 1 previous similar message [ 7782.366429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7782.373201] Lustre: Skipped 2 previous similar messages [ 7782.392389] Lustre: 172636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7789.779920] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7790.244379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7790.269991] Lustre: Skipped 1 previous similar message [ 7790.296351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7790.509609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7803.296302] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7811.762595] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7812.042547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7818.400553] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7818.724959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7826.411947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.419323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7826.447883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.493452] Lustre: Skipped 1 previous similar message [ 7827.887856] Lustre: server umount lustre-OST0000 complete [ 7840.054198] Lustre: server umount lustre-MDT0000 complete [ 7841.768502] LustreError: 170677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7841.822747] LustreError: 170677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 7845.057315] LustreError: 172635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408872 with bad export cookie 7170016517861059630 [ 7845.059819] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.078101] LustreError: 172635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7845.583465] Lustre: server umount lustre-MDT0001 complete [ 7859.365356] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:41:24 (1786408884) [ 7868.015924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.199347] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7886.501842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.296205] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7904.212813] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7911.403978] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7919.988650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.067650] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7936.096938] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7955.935856] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:43:00 (1786408980) [ 7962.600549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7962.610327] LustreError: Skipped 1 previous similar message [ 7962.618661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.634412] Lustre: Skipped 3 previous similar messages [ 7962.640975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7962.647931] Lustre: Skipped 3 previous similar messages [ 7967.088633] Lustre: server umount lustre-OST0000 complete [ 7978.958726] Lustre: server umount lustre-MDT0000 complete [ 7983.949121] LustreError: 175035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409011 with bad export cookie 7170016517861060512 [ 7983.957328] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.965958] LustreError: 175035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7984.397384] Lustre: server umount lustre-MDT0001 complete [ 7997.682618] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8001.857525] Key type lgssc unregistered [ 8002.406632] LNet: 178799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8002.414722] LNetError: 178799:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8002.426948] LNet: Removed LNI 192.168.204.128@tcp [ 8003.301256] Key type .llcrypt unregistered [ 8003.302727] Key type ._llcrypt unregistered [ 8028.342878] Key type ._llcrypt registered [ 8028.345888] Key type .llcrypt registered [ 8028.489843] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8030.671993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8030.827217] alg: No test for adler32 (adler32-zlib) [ 8032.559595] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 8033.093479] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8034.903484] Key type lgssc registered [ 8037.288786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8048.100891] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8048.120323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8063.647414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8068.767489] LustreError: 180046:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 8073.889475] LustreError: 180046:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8073.910153] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8073.920297] LustreError: MGC192.168.204.128@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8073.937384] LustreError: 180046:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8073.952532] LustreError: 180046:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8073.964195] LustreError: 180046:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8074.124352] Lustre: server umount lustre-OST0000 complete [ 8074.126051] LustreError: 180046:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8083.674064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.169837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8091.263315] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8095.715350] LustreError: 180546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8095.759309] LustreError: 180546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8100.847447] LustreError: 180545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.319863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.722699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8107.447264] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8116.857245] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8124.148357] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8132.405947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8132.566791] LustreError: 182363:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8132.574607] Lustre: 182363:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.128@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8132.761786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.964912] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8151.853208] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8157.665328] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.675677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.689851] Lustre: Skipped 1 previous similar message [ 8161.248963] Lustre: server umount lustre-MDT0000 complete [ 8162.789085] LustreError: 180546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8162.839669] LustreError: 180546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8165.592136] LustreError: 180527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409193 with bad export cookie 13114287352215590477 [ 8165.598052] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8165.610952] LustreError: 180527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8166.103257] Lustre: server umount lustre-MDT0001 complete [ 8181.007050] Lustre: server umount lustre-OST0000 complete [ 8192.104532] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8206.328424] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8254.871984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.405262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8255.455279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8255.548949] Lustre: lustre-MDT0000: new disk, initializing [ 8255.657860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8255.678870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8261.363187] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8272.878884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.981231] Lustre: 185894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8273.079380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8273.089705] Lustre: Skipped 1 previous similar message [ 8273.206168] Lustre: lustre-MDT0001: new disk, initializing [ 8273.388794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8273.451354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8273.462101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8279.732523] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8290.989791] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8297.392918] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8305.727634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8306.305663] Lustre: lustre-OST0000: new disk, initializing [ 8306.308970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8306.315616] Lustre: 187863:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8306.450600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.003091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8312.027846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8312.240261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8314.847606] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8326.703646] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8333.307716] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8333.602290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8338.851969] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8339.160747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8343.011603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.027367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.038775] Lustre: Skipped 1 previous similar message [ 8343.044906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8343.049953] Lustre: Skipped 3 previous similar messages [ 8348.128825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.137186] Lustre: Skipped 1 previous similar message [ 8353.252259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8356.320870] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8356.598228] Lustre: server umount lustre-OST0000 complete [ 8363.494333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.511108] Lustre: Skipped 1 previous similar message [ 8363.520548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8363.527513] Lustre: Skipped 2 previous similar messages [ 8367.886964] Lustre: server umount lustre-MDT0000 complete [ 8368.610594] LustreError: 185906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.633862] LustreError: 185906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8372.072499] LustreError: 185887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409399 with bad export cookie 13114287352215591191 [ 8372.082643] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8372.092977] LustreError: 185887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8372.602805] Lustre: server umount lustre-MDT0001 complete [ 8382.456811] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:50:08 (1786409408) [ 8399.694825] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8402.334468] Key type lgssc unregistered [ 8402.637322] LNet: 190812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8402.647021] LNetError: 190812:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8402.679514] LNet: Removed LNI 192.168.204.128@tcp [ 8403.593895] Key type .llcrypt unregistered [ 8403.597802] Key type ._llcrypt unregistered [ 8423.988219] Key type ._llcrypt registered [ 8423.996504] Key type .llcrypt registered [ 8424.209962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8426.033922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8426.069719] alg: No test for adler32 (adler32-zlib) [ 8427.220798] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 8427.490481] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8429.175203] Key type lgssc registered [ 8431.175423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8441.080809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8441.104424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.741438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8448.640637] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8453.092055] LustreError: 192032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8453.118963] LustreError: 192032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8458.218301] LustreError: 192031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.555790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.951736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8463.876169] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8473.398099] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8480.678261] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8489.815983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8490.057364] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8490.064519] LustreError: 193895:0:(llog.c:1696:llog_backup()) MGC192.168.204.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8490.072249] Lustre: 193895:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.128@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8490.214042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8497.931308] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8507.240818] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8510.433879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8510.441873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.453689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8514.023128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.030863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8514.040572] Lustre: Skipped 1 previous similar message [ 8514.060733] Lustre: Skipped 2 previous similar messages [ 8516.577270] Lustre: server umount lustre-MDT0000 complete [ 8519.151541] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8519.190407] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8522.073823] LustreError: 192011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409549 with bad export cookie 8701184488011656501 [ 8522.089973] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.096031] LustreError: 192011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8522.643468] Lustre: server umount lustre-MDT0001 complete [ 8538.160968] Lustre: server umount lustre-OST0000 complete [ 8549.507166] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8563.814884] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8606.094031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8606.603996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8606.661160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8606.794559] Lustre: lustre-MDT0000: new disk, initializing [ 8606.970795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.019438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8616.613385] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8634.088574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.367275] Lustre: 197440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8634.468988] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8634.474092] Lustre: Skipped 1 previous similar message [ 8634.484023] LustreError: 198224:0:(llog.c:1696:llog_backup()) MGC192.168.204.128@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8634.507875] LustreError: 198224:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8634.522249] Lustre: 198224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.128@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8634.576414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8634.588264] Lustre: Skipped 1 previous similar message [ 8634.718161] Lustre: lustre-MDT0001: new disk, initializing [ 8634.891815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8634.958871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8634.980046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8643.493537] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8660.200905] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8669.918323] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8681.112182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8681.806209] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8681.811170] Lustre: Skipped 1 previous similar message [ 8681.815922] LustreError: 199390:0:(llog.c:1696:llog_backup()) MGC192.168.204.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8681.828038] LustreError: 199390:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8681.839489] Lustre: 199390:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.128@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8681.851236] Lustre: 199390:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8681.883694] Lustre: lustre-OST0000: new disk, initializing [ 8681.893234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8681.905614] Lustre: 199415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8682.093958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.893375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8689.946844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8690.188796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8691.860677] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8702.605497] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8708.357168] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8708.653412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8714.582614] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8714.794613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8720.865884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8720.874886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8720.883257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8720.897550] Lustre: Skipped 1 previous similar message [ 8723.927827] Lustre: server umount lustre-OST0000 complete [ 8731.111712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8731.119580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8731.139690] Lustre: Skipped 1 previous similar message [ 8731.147362] Lustre: Skipped 1 previous similar message [ 8736.224287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8736.234565] Lustre: Skipped 1 previous similar message [ 8743.391130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8743.550805] Lustre: server umount lustre-MDT0000 complete [ 8745.440688] LustreError: 197451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8745.455833] LustreError: 197451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8747.691450] LustreError: 197430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409775 with bad export cookie 8701184488011657215 [ 8747.697224] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8747.709112] LustreError: 197430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8748.111979] Lustre: server umount lustre-MDT0001 complete [ 8758.894768] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:56:24 (1786409784) [ 8778.358847] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8791.317147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.669564] LustreError: 202544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.734377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.144901] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8801.252197] LustreError: 202545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8801.264693] LustreError: 202545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8806.441904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.821812] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8814.703289] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8822.308403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8830.424372] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8840.851325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8841.053451] Lustre: lustre-OST0001: new disk, initializing [ 8841.058643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8841.070808] Lustre: 204851:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8841.194776] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8841.210735] Lustre: Skipped 2 previous similar messages [ 8848.255947] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8849.059119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8849.073668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8849.182548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8860.956922] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8870.100917] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8872.906583] Lustre: DEBUG MARKER: ADD OST3 [ 8879.459917] loop: module loaded [ 8879.625327] loop0: detected capacity change from 0 to 204800000 [ 8880.105224] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8880.139237] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8880.208076] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8888.734489] loop0: detected capacity change from 0 to 204800000 [ 8888.867271] Lustre: DEBUG MARKER: STOP OST3 [ 8896.988602] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8897.602054] Lustre: lustre-OST0003: new disk, initializing [ 8897.614962] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8897.623788] Lustre: 206895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8899.697240] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8899.839883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8899.848736] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8899.914135] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8901.727065] LustreError: 207015:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8901.823487] Lustre: Failing over lustre-OST0003 [ 8902.058882] Lustre: server umount lustre-OST0003 complete [ 8904.850846] LustreError: 205236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.874699] LustreError: 205236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8905.230357] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.257840] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8905.282758] LustreError: Skipped 1 previous similar message [ 8908.924434] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8923.613503] Lustre: DEBUG MARKER: START OST3 again [ 8932.879559] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8933.361268] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8933.382733] Lustre: Skipped 1 previous similar message [ 8933.408708] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8935.088944] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8935.454310] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8935.454881] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8942.621761] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8968.166885] Lustre: Failing over lustre-OST0003 [ 8968.275297] Lustre: server umount lustre-OST0003 complete [ 8969.227938] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.246219] LustreError: 204032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8969.261418] Lustre: Skipped 2 previous similar messages [ 8969.329738] LustreError: 204032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 8989.669269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8989.672579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8989.678466] Lustre: Skipped 3 previous similar messages [ 8989.684209] Lustre: Skipped 5 previous similar messages [ 8994.470896] Lustre: server umount lustre-MDT0000 complete [ 8999.045264] LustreError: 202524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410026 with bad export cookie 8701184488011658097 [ 8999.047577] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8999.080566] LustreError: 202524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9005.675565] Lustre: server umount lustre-MDT0001 complete [ 9010.599606] Lustre: server umount lustre-OST0000 complete [ 9016.591137] Lustre: server umount lustre-OST0001 complete [ 9025.877837] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 9039.376068] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9083.968679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.350282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9084.386734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9084.470957] Lustre: lustre-MDT0000: new disk, initializing [ 9084.585188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.603109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9089.955964] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9102.617030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.746471] Lustre: 212487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9102.816264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9102.827415] Lustre: Skipped 1 previous similar message [ 9102.908410] Lustre: lustre-MDT0001: new disk, initializing [ 9103.149798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9103.178198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9108.809880] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9120.311278] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9127.670265] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9136.394136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9136.667350] Lustre: lustre-OST0000: new disk, initializing [ 9136.671611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9136.680341] Lustre: 214458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9138.074129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9138.094337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9138.392958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9144.727634] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9154.054971] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9159.078897] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9159.331650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9164.606640] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9164.823459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9168.866344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9168.875658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.896817] Lustre: Skipped 2 previous similar messages [ 9168.910177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9168.922750] Lustre: Skipped 4 previous similar messages [ 9173.333932] Lustre: server umount lustre-OST0000 complete [ 9184.376710] Lustre: server umount lustre-MDT0000 complete [ 9184.739041] LustreError: 214405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9184.760575] LustreError: 214405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9189.272855] LustreError: 212481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410217 with bad export cookie 8701184488011660386 [ 9189.282325] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9189.557041] Lustre: server umount lustre-MDT0001 complete [ 9193.928901] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:03:39 (1786410219) [ 9216.002925] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 9230.906175] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9275.422941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.683554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9275.729581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9275.826398] Lustre: lustre-MDT0000: new disk, initializing [ 9275.911671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9281.093467] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9293.042779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.202267] Lustre: 219085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9293.565739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9299.170934] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9309.584804] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9316.820491] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9324.523455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9324.879675] Lustre: 221057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9326.127248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9326.312167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9334.534235] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9346.768452] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9352.313390] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9352.566971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9358.149025] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9358.411543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9362.402898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9362.416028] LustreError: Skipped 1 previous similar message [ 9362.417929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.417940] Lustre: Skipped 3 previous similar messages [ 9362.426993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9362.468260] Lustre: Skipped 5 previous similar messages [ 9375.199125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9375.413324] Lustre: server umount lustre-OST0000 complete [ 9380.835978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9385.961059] Lustre: server umount lustre-MDT0000 complete [ 9388.008844] LustreError: 219093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9388.052411] LustreError: 219093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9390.637585] LustreError: 219078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410418 with bad export cookie 8701184488011661268 [ 9390.666333] LustreError: 219078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9390.679563] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.115852] Lustre: server umount lustre-MDT0001 complete [ 9401.152029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9401.809310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.826369] Lustre: Skipped 5 previous similar messages [ 9407.995582] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9419.591678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.572713] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9434.122252] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9440.002521] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9447.523925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9456.086717] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9465.240801] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9485.282647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9485.294059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9485.301823] Lustre: Skipped 7 previous similar messages [ 9486.147491] Lustre: server umount lustre-OST0000 complete [ 9491.427595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.438183] Lustre: Skipped 4 previous similar messages [ 9501.798101] Lustre: server umount lustre-MDT0000 complete [ 9506.560229] LustreError: 226177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410534 with bad export cookie 8701184488011662150 [ 9506.573747] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9506.583035] LustreError: 226177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9506.914585] Lustre: server umount lustre-MDT0001 complete [ 9613.320871] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9617.530745] Key type lgssc unregistered [ 9617.836852] LNet: 226962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9617.845966] LNetError: 226962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9617.868946] LNet: Removed LNI 192.168.204.128@tcp [ 9618.785479] Key type .llcrypt unregistered [ 9618.791160] Key type ._llcrypt unregistered [ 9650.636364] Key type ._llcrypt registered [ 9650.638986] Key type .llcrypt registered [ 9650.752961] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 9671.091619] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9673.422391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9673.563435] alg: No test for adler32 (adler32-zlib) [ 9674.737334] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 9674.984124] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9676.703204] Key type lgssc registered [ 9678.085555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9719.229579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9719.289880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9720.751952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9720.777876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9720.844987] Lustre: lustre-MDT0000: new disk, initializing [ 9720.924837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.940314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9726.826893] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9738.580821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.696675] Lustre: 230321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9738.769471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9738.775633] Lustre: Skipped 1 previous similar message [ 9738.875461] Lustre: lustre-MDT0001: new disk, initializing [ 9739.107550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9739.205186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9739.230076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9744.986709] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9756.894454] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9765.516454] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9774.499232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9774.744888] Lustre: lustre-OST0000: new disk, initializing [ 9774.749713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9774.757266] Lustre: 232290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9774.872239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9781.814899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9781.830793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9781.955132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9783.630423] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9795.011967] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9801.992980] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9802.306951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9810.244973] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9810.712657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9818.083147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.086659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9818.089097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9818.099498] Lustre: Skipped 1 previous similar message [ 9818.132201] LustreError: Skipped 1 previous similar message [ 9819.848092] Lustre: server umount lustre-OST0000 complete [ 9826.273584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9826.288287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.311764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.322731] Lustre: Skipped 1 previous similar message [ 9828.320614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9828.325335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.896176] Lustre: server umount lustre-MDT0000 complete [ 9835.575775] LustreError: 232288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410863 with bad export cookie 15901693044411329927 [ 9835.590947] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9835.593571] LustreError: 232288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9836.116906] Lustre: server umount lustre-MDT0001 complete [ 9864.367996] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9878.522688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.176264] LustreError: 235246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.205918] LustreError: 235246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9879.316358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9884.639948] LustreError: 235247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9885.689585] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9889.760667] LustreError: 235246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.883280] LustreError: 235247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.070861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.764811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9904.601975] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9908.611217] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9916.845954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9917.291336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9925.113437] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9935.711835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9935.916744] Lustre: lustre-OST0001: new disk, initializing [ 9935.920076] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9935.926446] Lustre: 237556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9936.049310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9943.664711] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9946.303259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9946.310660] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9946.419566] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9955.197481] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9963.357981] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9981.024466] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:16:46 (1786411006) [ 9984.993170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9984.998372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.010621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9985.016266] Lustre: Skipped 1 previous similar message [ 9992.677061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9992.688033] Lustre: Skipped 7 previous similar messages [ 9998.303136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9998.589732] Lustre: server umount lustre-MDT0000 complete [10002.924854] LustreError: 238214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.963639] LustreError: 238214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10003.178901] LustreError: 235228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411031 with bad export cookie 15901693044411330809 [10003.181902] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10003.195029] LustreError: 235228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10003.678420] Lustre: server umount lustre-MDT0001 complete [10019.234349] Lustre: server umount lustre-OST0000 complete [10023.393085] Lustre: 228543:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786411035/real 1786411035] req@ffff9aa50b00f480 x1873187388384768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786411051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10023.459783] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.494393] Lustre: Skipped 3 previous similar messages [10024.304841] Lustre: server umount lustre-OST0001 complete [10032.818995] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [10049.147321] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10093.950656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10094.247305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10094.274987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10094.336672] Lustre: lustre-MDT0000: new disk, initializing [10094.408435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.460756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10100.425647] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10114.404602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10114.676852] Lustre: 241528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10114.854502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10114.857693] Lustre: Skipped 1 previous similar message [10114.940913] Lustre: lustre-MDT0001: new disk, initializing [10115.196629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10115.209436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10122.460512] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10134.112325] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10140.763719] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10148.333490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10148.619247] Lustre: lustre-OST0000: new disk, initializing [10148.622516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10148.628522] Lustre: 243501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10150.357131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10150.385505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10150.625702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10158.038531] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10169.559281] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10175.394966] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10175.947267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10180.233302] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10180.650325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10184.687265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10184.707137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10184.712190] Lustre: Skipped 4 previous similar messages [10189.578818] Lustre: server umount lustre-OST0000 complete [10200.987803] Lustre: server umount lustre-MDT0000 complete [10204.130494] LustreError: 242373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.170448] LustreError: 242373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10205.367854] LustreError: 241519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411233 with bad export cookie 15901693044411332944 [10205.374993] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10205.391052] LustreError: 241519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10205.752706] Lustre: server umount lustre-MDT0001 complete [10215.709509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.152889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10216.164497] Lustre: Skipped 2 previous similar messages [10221.026667] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10221.538509] LustreError: 245717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.569683] LustreError: 245717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10230.212213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.047552] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10245.927094] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10252.464285] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10260.118205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10260.467754] LustreError: 247540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.502397] LustreError: 247540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10267.233330] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10274.968708] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10423.777162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10423.787627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10423.809167] Lustre: Skipped 4 previous similar messages [10423.819971] Lustre: Skipped 6 previous similar messages [10427.557824] Lustre: server umount lustre-OST0000 complete [10437.880373] Lustre: server umount lustre-MDT0000 complete [10439.142907] LustreError: 245718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.163307] LustreError: 245718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10441.102275] LustreError: 246453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411468 with bad export cookie 15901693044411333826 [10441.102275] LustreError: 245700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411468 with bad export cookie 15901693044411333826 [10441.102936] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10441.419345] Lustre: server umount lustre-MDT0001 complete [10546.341785] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10548.972744] Key type lgssc unregistered [10549.261823] LNet: 249327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10549.273579] LNetError: 249327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10549.289745] LNet: Removed LNI 192.168.204.128@tcp [10550.155195] Key type .llcrypt unregistered [10550.158443] Key type ._llcrypt unregistered [10577.328486] Key type ._llcrypt registered [10577.330480] Key type .llcrypt registered [10577.429460] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [10595.686279] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10598.354787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10598.498975] alg: No test for adler32 (adler32-zlib) [10600.268845] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [10600.692677] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [10602.535264] Key type lgssc registered [10603.709874] Lustre: Echo OBD driver; http://www.lustre.org/ [10660.323486] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10671.309793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10671.352067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.648252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10672.679176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10672.762087] Lustre: lustre-MDT0000: new disk, initializing [10672.843737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10672.865938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10678.119198] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10688.440525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10688.572928] Lustre: 253401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10688.683731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10688.689523] Lustre: Skipped 1 previous similar message [10688.831987] Lustre: lustre-MDT0001: new disk, initializing [10688.968538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10689.019941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10689.036108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10693.758709] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10699.102279] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10707.530087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10707.947903] Lustre: lustre-OST0000: new disk, initializing [10707.953635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10707.961607] Lustre: 255031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10708.073363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10709.200521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10709.234740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10709.558981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10715.771454] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10727.406921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10727.662679] Lustre: lustre-OST0001: new disk, initializing [10727.674790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10727.683699] Lustre: 255905:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10727.892243] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10734.779604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10734.786885] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10734.987857] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10735.548851] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10748.863654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10759.371727] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10769.764264] Lustre: server umount lustre-MDT0000 complete [10770.927629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.951169] Lustre: Skipped 1 previous similar message [10770.963229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10774.399655] LustreError: 253394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411802 with bad export cookie 12739763633469532908 [10774.404566] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10774.415881] LustreError: 253394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10774.802724] Lustre: server umount lustre-MDT0001 complete [10789.640699] Lustre: server umount lustre-OST0000 complete [10791.583165] Lustre: 250907:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786411803/real 1786411803] req@ffff9aa50bc7a680 x1873188359321344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786411819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10791.667287] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.699621] Lustre: Skipped 2 previous similar messages [10794.657902] Lustre: server umount lustre-OST0001 complete [10811.037819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10822.555683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10833.823379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10843.690611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10865.810888] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10876.384301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.401633] Lustre: lustre-MDT0000: reset Object Index mappings [10876.808157] LustreError: 260914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.826868] LustreError: 260914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10876.902033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10882.027091] LustreError: 260915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.150761] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10886.132873] LustreError: 260914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.244355] LustreError: 260915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.287775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10893.307173] Lustre: lustre-MDT0001: reset Object Index mappings [10893.703337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10898.507939] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10901.862171] Lustre: 262057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10908.999245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10917.998712] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10918.702791] LustreError: 262409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.786828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10927.084164] LustreError: 262412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.106238] LustreError: 262412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10927.591880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10927.856602] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10927.868836] Lustre: Skipped 1 previous similar message [10933.253674] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10934.828764] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10943.015880] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10947.060158] Lustre: 263928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10953.700744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10953.706675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10953.709111] Lustre: Skipped 3 previous similar messages [10953.712524] Lustre: Skipped 2 previous similar messages [10954.667647] Lustre: server umount lustre-MDT0000 complete [10958.827859] LustreError: 260915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.864606] LustreError: 260915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10959.541640] LustreError: 263930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411987 with bad export cookie 12739763633469539698 [10959.549221] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10959.569337] LustreError: 263930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10959.942260] Lustre: server umount lustre-MDT0001 complete [10974.201448] Lustre: server umount lustre-OST0000 complete [10988.887461] Lustre: server umount lustre-OST0001 complete [11013.368605] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:33:59 (1786412039) [11033.574477] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [11045.865731] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11087.479126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11087.941193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11087.993668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11088.086321] Lustre: lustre-MDT0000: new disk, initializing [11088.191984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11088.217360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11093.943530] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11107.739360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11107.895988] Lustre: 268185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11107.947468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11107.951694] Lustre: Skipped 1 previous similar message [11108.045348] Lustre: lustre-MDT0001: new disk, initializing [11108.244712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11108.257052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11114.066264] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11126.065504] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11132.859290] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11140.264057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11140.659425] Lustre: lustre-OST0000: new disk, initializing [11140.667325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11140.674642] Lustre: 270157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11141.991593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11142.002508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11142.224058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11148.158633] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11161.476332] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11167.468740] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11167.680509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11173.484211] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11173.693637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11177.964982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11177.968550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11177.976729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11177.976737] Lustre: Skipped 1 previous similar message [11177.986412] Lustre: Skipped 1 previous similar message [11178.002508] LustreError: Skipped 1 previous similar message [11183.077462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11183.085419] Lustre: Skipped 2 previous similar messages [11188.196495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11188.205627] Lustre: Skipped 1 previous similar message [11190.239882] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11190.407781] Lustre: server umount lustre-OST0000 complete [11195.364045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11195.376531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.404208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11201.868886] Lustre: server umount lustre-MDT0000 complete [11203.573755] LustreError: 268198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.615186] LustreError: 268198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11207.144310] LustreError: 270156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412235 with bad export cookie 12739763633469541791 [11207.169375] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11207.171416] LustreError: 270156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11207.641568] Lustre: server umount lustre-MDT0001 complete [11231.323284] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11243.229616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11243.601791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11243.608639] Lustre: Skipped 2 previous similar messages [11249.323965] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11258.768681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11264.598321] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11267.880344] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11274.830253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11283.742723] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11294.352899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11294.576449] Lustre: lustre-OST0001: new disk, initializing [11294.600520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11294.614311] Lustre: 275405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11296.453711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11296.496926] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11296.802723] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11303.172974] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11315.168108] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11320.090121] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11327.188346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11327.250878] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11336.164707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11336.171623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11336.185509] Lustre: Skipped 1 previous similar message [11336.194585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.202736] Lustre: Skipped 2 previous similar messages [11341.384988] Lustre: server umount lustre-MDT0000 complete [11342.825936] LustreError: 274571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11342.872847] LustreError: 274571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11346.256687] LustreError: 273829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412374 with bad export cookie 12739763633469542673 [11346.267496] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11346.273913] LustreError: 273829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11346.660709] Lustre: server umount lustre-MDT0001 complete [11361.986951] Lustre: server umount lustre-OST0000 complete [11363.815326] Lustre: 250904:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786412375/real 1786412375] req@ffff9aa50181ed80 x1873188359486976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786412391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11363.890171] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11363.938246] Lustre: Skipped 3 previous similar messages [11366.638843] Lustre: server umount lustre-OST0001 complete [11387.215585] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11399.797234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11400.467499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11400.485868] Lustre: Skipped 3 previous similar messages [11406.977933] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11418.134372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11424.797398] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11428.275154] Lustre: 279601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11435.754659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11442.862928] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11445.548970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11451.636240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11457.041955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11458.730892] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11466.959616] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11470.829452] Lustre: 281481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11478.071121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11478.134478] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11489.613773] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:41:55 (1786412515) [11505.635838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11505.647687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.659614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11505.665321] Lustre: Skipped 4 previous similar messages [11510.760855] Lustre: server umount lustre-MDT0000 complete [11513.325482] LustreError: 278455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11513.353579] LustreError: 278455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [11515.603144] LustreError: 278439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412543 with bad export cookie 12739763633469549015 [11515.625994] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11515.626680] LustreError: 278439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11522.254401] Lustre: server umount lustre-MDT0001 complete [11533.326509] Lustre: server umount lustre-OST0000 complete [11537.863558] Lustre: server umount lustre-OST0001 complete [11546.459799] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [11564.030538] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11625.745920] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11639.042054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.293483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11639.317909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11639.403909] Lustre: lustre-MDT0000: new disk, initializing [11639.513452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11644.814774] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11658.089652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.255330] Lustre: 285765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11658.556530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11658.569965] Lustre: Skipped 4 previous similar messages [11658.663116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11663.903926] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11677.034221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11677.357729] Lustre: 287396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11679.413946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11684.989099] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11697.243285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11697.488451] Lustre: 288271:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11699.654741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11699.678366] Lustre: Skipped 1 previous similar message [11699.887882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11705.394692] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11716.825913] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11721.233141] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11721.239584] Lustre: Skipped 1 previous similar message [11738.027382] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:46:03 (1786412763) [11745.762852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11745.770922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11745.780207] Lustre: Skipped 5 previous similar messages [11745.785698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11745.793392] Lustre: Skipped 8 previous similar messages [11747.647883] Lustre: server umount lustre-MDT0000 complete [11752.046621] LustreError: 285758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412779 with bad export cookie 12739763633469556883 [11752.061475] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11752.459832] Lustre: server umount lustre-MDT0001 complete [11766.380623] Lustre: server umount lustre-OST0000 complete [11780.886846] Lustre: server umount lustre-OST0001 complete [11789.896488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11790.209313] LustreError: 290518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11790.223486] LustreError: 290518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [11795.148962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11804.842487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11811.063507] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11819.543895] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11826.423701] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11833.498133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11841.878674] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11851.222758] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11870.012767] loop1: detected capacity change from 0 to 204800000 [11870.324828] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11870.365855] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11870.389637] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11876.022900] loop1: detected capacity change from 0 to 204800000 [11883.385068] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11883.504214] LustreError: 290507:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12434: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11883.542991] LustreError: 290507:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12434 [11883.561214] LustreError: 290507:0:(mgs_handler.c:517:mgs_target_reg()) fs12434-MDT0001: Error checking target index: rc = -13 [11883.593129] LustreError: fs12434-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11883.606790] LustreError: 294241:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11883.613851] LustreError: 294241:0:(tgt_mount.c:2129:server_put_super()) no obd fs12434-MDT0001 [11883.621452] LustreError: 294241:0:(tgt_mount.c:116:server_deregister_mount()) fs12434-MDT0001 not registered [11883.700567] Lustre: server umount fs12434-MDT0001 complete [11883.706507] LustreError: 294241:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11894.613624] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11894.693621] Lustre: Setting parameter fs12434-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12434-MDT0001 [11894.729412] Lustre: srv-fs12434-MDT0001: No data found on store. Initialize space. [11894.736321] Lustre: Skipped 4 previous similar messages [11894.860803] Lustre: fs12434-MDT0001: new disk, initializing [11894.862819] Lustre: Skipped 3 previous similar messages [11895.008911] Lustre: srv-fs12434-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11895.021471] Lustre: 294707:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12434-MDT0001-mdtlov: can't create new llog: rc = -115 [11895.034882] LustreError: 294707:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12434-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11900.534673] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11907.847947] Lustre: server umount fs12434-MDT0001 complete [11911.696296] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11925.325876] loop1: detected capacity change from 0 to 204800000 [11933.358599] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11933.894196] Lustre: 296277:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12434-MDT0001-mdtlov: can't create new llog: rc = -115 [11933.933490] LustreError: 296277:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12434-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11940.425279] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11946.065940] Lustre: server umount fs12434-MDT0001 complete [11958.754954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11958.772339] LustreError: Skipped 1 previous similar message [11958.779080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11958.793052] Lustre: Skipped 3 previous similar messages [11958.809208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11958.821780] Lustre: Skipped 4 previous similar messages [11973.098523] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11973.313104] Lustre: server umount lustre-MDT0000 complete [11983.096611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11983.263545] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11988.977217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11989.866362] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11999.154911] loop1: detected capacity change from 0 to 204800000 [12006.454570] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12006.790603] Lustre: 298725:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12434-MDT0001-mdtlov: can't create new llog: rc = -115 [12006.803434] LustreError: 298725:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12434-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12013.326862] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12018.651582] Lustre: server umount fs12434-MDT0001 complete [12044.773411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12057.055127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12057.245780] Lustre: server umount lustre-OST0000 complete [12068.598523] Lustre: server umount lustre-MDT0000 complete [12073.193965] LustreError: 290499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413101 with bad export cookie 12739763633469559571 [12073.218278] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12073.228656] LustreError: 290499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12073.634115] Lustre: server umount lustre-MDT0001 complete [12086.139128] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12088.998618] Key type lgssc unregistered [12089.271475] LNet: 301071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12089.279733] LNetError: 301071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12089.299737] LNet: Removed LNI 192.168.204.128@tcp [12090.108286] Key type .llcrypt unregistered [12090.113708] Key type ._llcrypt unregistered [12113.772883] Key type ._llcrypt registered [12113.776062] Key type .llcrypt registered [12113.997735] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12117.206093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12117.318799] alg: No test for adler32 (adler32-zlib) [12118.522071] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [12118.913284] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12120.719164] Key type lgssc registered [12122.475353] Lustre: Echo OBD driver; http://www.lustre.org/ [12133.159027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12133.193588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12134.965497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12141.102266] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12145.138543] LustreError: 302347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12145.171090] LustreError: 302347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12150.245111] LustreError: 302348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.635515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12152.437918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12159.349818] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12169.450311] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12176.485148] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12184.525484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12184.945335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12193.718625] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12198.274512] LustreError: 304180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.372502] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12203.533856] LustreError: 304180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.665991] LustreError: 304178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.687234] LustreError: 304178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12220.949428] LustreError: 304708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12221.001291] LustreError: 304708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12231.674947] loop1: detected capacity change from 0 to 204800000 [12231.987130] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12232.027457] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12232.083907] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12238.392510] loop1: detected capacity change from 0 to 204800000 [12241.389748] LustreError: 304178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12241.400471] LustreError: 304178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [12245.669772] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12245.805114] Lustre: fs970-OST0000: new disk, initializing [12245.808371] Lustre: srv-fs970-OST0000: No data found on store. Initialize space. [12245.812635] Lustre: 306102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12245.883106] Lustre: fs970-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12254.829984] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12265.316453] Lustre: server umount fs970-OST0000 complete [12277.222986] LustreError: 304708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.229611] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12277.260984] LustreError: 304708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [12282.211848] loop1: detected capacity change from 0 to 204800000 [12282.356491] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12289.106893] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12289.351796] Lustre: fs970-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12297.737696] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12308.694584] loop2: detected capacity change from 0 to 204800000 [12309.059782] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12309.081769] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12309.091625] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12315.314842] loop2: detected capacity change from 0 to 204800000 [12322.503287] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12322.716383] LustreError: fs970-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12322.754451] LustreError: fs970-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12322.810111] LustreError: 309143:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12322.822946] LustreError: 309143:0:(tgt_mount.c:2129:server_put_super()) no obd fs970-OST0001 [12322.830355] LustreError: 309143:0:(tgt_mount.c:116:server_deregister_mount()) fs970-OST0001 not registered [12322.932600] Lustre: server umount fs970-OST0001 complete [12322.934900] LustreError: 309143:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [12336.865285] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12337.032857] Lustre: fs970-OST0001: new disk, initializing [12337.036511] Lustre: srv-fs970-OST0001: No data found on store. Initialize space. [12337.049136] Lustre: 309694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12337.154604] Lustre: fs970-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12343.796855] LustreError: 304178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.826652] LustreError: 304178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [12345.824451] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12357.469814] Lustre: server umount fs970-OST0000 complete [12369.143582] Lustre: server umount fs970-OST0001 complete [12384.039392] loop1: detected capacity change from 0 to 204800000 [12390.455809] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12390.738402] Lustre: fs970-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12399.037792] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12407.798696] loop2: detected capacity change from 0 to 204800000 [12414.962689] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12422.649634] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12430.840660] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12434.022922] Lustre: server umount fs970-OST0000 complete [12435.958940] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12444.111951] Lustre: server umount fs970-OST0001 complete [12459.488684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12459.497620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12459.514537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12461.546596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.561834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12461.583676] Lustre: Skipped 1 previous similar message [12466.665732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12471.263150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12471.432894] Lustre: server umount lustre-OST0000 complete [12476.901697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12476.922260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12476.936504] Lustre: Skipped 2 previous similar messages [12482.019281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12482.040561] Lustre: Skipped 1 previous similar message [12482.625911] Lustre: server umount lustre-MDT0000 complete [12486.877047] LustreError: 302329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413514 with bad export cookie 9312140025906750982 [12486.887662] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12486.921247] LustreError: 302329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12487.378658] Lustre: server umount lustre-MDT0001 complete [12500.812214] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12505.118949] Key type lgssc unregistered [12505.601918] LNet: 315622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12505.612319] LNetError: 315622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12506.662735] LNet: Removed LNI 192.168.204.128@tcp [12508.379277] Key type .llcrypt unregistered [12508.383963] Key type ._llcrypt unregistered [12545.286700] Key type ._llcrypt registered [12545.289306] Key type .llcrypt registered [12545.393659] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [12562.990463] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12564.658791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12564.716319] alg: No test for adler32 (adler32-zlib) [12565.865811] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [12566.175411] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12567.871548] Key type lgssc registered [12569.770949] Lustre: Echo OBD driver; http://www.lustre.org/ [12614.411299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12614.440966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12615.759849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12615.825913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12615.965326] Lustre: lustre-MDT0000: new disk, initializing [12616.160408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12616.183688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12621.782910] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12634.774371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12635.109400] Lustre: 319492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12635.220671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12635.223967] Lustre: Skipped 1 previous similar message [12635.291480] Lustre: lustre-MDT0001: new disk, initializing [12635.358267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12635.410644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12635.429280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12641.766271] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12653.253063] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12659.139709] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12667.325182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12667.787512] Lustre: lustre-OST0000: new disk, initializing [12667.792682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12667.799227] Lustre: 321472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12667.870351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12675.608451] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12677.772154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12677.785620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12677.949523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12687.283979] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12690.042208] Lustre: Mounted lustre-client [12691.177894] Lustre: Unmounted lustre-client [12703.712670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12703.714515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12703.720759] LustreError: Skipped 1 previous similar message [12703.722893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12703.754270] Lustre: Skipped 1 previous similar message [12705.260973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12705.270818] Lustre: Skipped 1 previous similar message [12707.622422] Lustre: server umount lustre-OST0000 complete [12712.421191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12712.428506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.444100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12715.492454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12715.498861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12715.518997] Lustre: Skipped 1 previous similar message [12718.432855] Lustre: server umount lustre-MDT0000 complete [12722.496488] LustreError: 319484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413750 with bad export cookie 1787354182019279503 [12722.503538] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12722.517563] LustreError: 319484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12722.815867] Lustre: server umount lustre-MDT0001 complete [12738.844374] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12742.754638] Key type lgssc unregistered [12749.408285] LNet: 323638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12749.420451] LNetError: 323638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12822.118864] LNet: Removed LNI 192.168.204.128@tcp [12823.130484] Key type .llcrypt unregistered [12823.133203] Key type ._llcrypt unregistered [12844.941984] Key type ._llcrypt registered [12844.945823] Key type .llcrypt registered [12845.137700] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [12864.941943] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12867.217194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12867.311327] alg: No test for adler32 (adler32-zlib) [12868.573522] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [12868.932378] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12870.655223] Key type lgssc registered [12872.367604] Lustre: Echo OBD driver; http://www.lustre.org/ [12908.632462] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:05:33 (1786413933) [12916.360219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12916.382452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12917.649332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12917.669404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12917.766428] Lustre: lustre-MDT0000: new disk, initializing [12917.869828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12917.904259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12924.369834] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12937.250584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12937.488735] Lustre: 326859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12937.594335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12937.598363] Lustre: Skipped 1 previous similar message [12937.693201] Lustre: lustre-MDT0001: new disk, initializing [12937.797973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12937.838040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12937.852760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12943.634706] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12953.868242] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12960.526615] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12968.351490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12968.683866] Lustre: lustre-OST0000: new disk, initializing [12968.694977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12968.713287] Lustre: 328829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12968.846729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12975.499602] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12976.254924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12976.269124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12976.439147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12986.631402] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13007.340604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13007.344610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13007.349896] LustreError: Skipped 1 previous similar message [13007.358775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13007.370465] Lustre: Skipped 1 previous similar message [13007.390662] Lustre: Skipped 1 previous similar message [13010.618668] Lustre: server umount lustre-OST0000 complete [13014.504466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13014.520566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13014.547716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13017.574362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13017.574694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13020.842664] Lustre: server umount lustre-MDT0000 complete [13024.303603] LustreError: 326852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414052 with bad export cookie 9994930647977631275 [13024.321780] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13024.601925] Lustre: server umount lustre-MDT0001 complete [13036.991629] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [13040.357833] Key type lgssc unregistered [13040.783996] LNet: 330736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13040.811219] LNetError: 330736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13041.894890] LNet: Removed LNI 192.168.204.128@tcp [13042.796364] Key type .llcrypt unregistered [13042.798654] Key type ._llcrypt unregistered [13066.487270] Key type ._llcrypt registered [13066.490754] Key type .llcrypt registered [13066.584817] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [13069.033939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13069.164087] alg: No test for adler32 (adler32-zlib) [13070.443913] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [13070.857418] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [13072.543131] Key type lgssc registered [13073.637902] Lustre: Echo OBD driver; http://www.lustre.org/ [13081.852363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13081.875566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13083.379182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13087.720124] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13093.858709] LustreError: 332010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13093.882918] LustreError: 332010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13097.242803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13097.580991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13102.661405] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13109.603685] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13115.508942] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13124.946860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13125.299029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13133.363439] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13143.844080] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13156.307498] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13158.165748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13163.498737] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.517177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13163.530153] Lustre: Skipped 2 previous similar messages [13167.219944] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [13168.613641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13168.626313] Lustre: Skipped 2 previous similar messages [13169.414348] Lustre: server umount lustre-MDT0000 complete [13172.330180] LustreError: 333853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.738490] LustreError: 332004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.785575] LustreError: 332004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13174.100879] LustreError: 331990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414201 with bad export cookie 14252375041400693802 [13174.108930] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13174.113439] LustreError: 331990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13177.435351] LustreError: 332006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13177.471069] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [13178.855772] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.876248] Lustre: Skipped 2 previous similar messages [13180.589756] Lustre: server umount lustre-MDT0001 complete [13186.203349] Lustre: server umount lustre-OST0000 complete [13196.874929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13203.378885] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13206.415364] Lustre: 336194:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.128@tcp [13208.750948] Lustre: server umount MGS complete [13219.822709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13220.546189] LustreError: 336766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.574881] LustreError: 336766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13220.671322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13226.899881] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13231.075027] LustreError: 336766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13231.091199] LustreError: 336766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13236.432409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13236.754855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13244.328939] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13256.537950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13256.982863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13266.504514] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13277.468879] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13280.382221] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13286.605346] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13288.829897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13294.764600] Lustre: server umount lustre-OST0000 complete [13301.216884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13301.221744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13301.233332] Lustre: Skipped 1 previous similar message [13301.249295] Lustre: Skipped 3 previous similar messages [13309.610534] Lustre: server umount lustre-MDT0000 complete [13311.459919] LustreError: 336762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13311.478976] LustreError: 336762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13312.481093] Lustre: 331322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786414324/real 1786414324] req@0000000074ed8e91 x1873190949291904/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786414340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13312.528092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13314.068581] LustreError: 338226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414341 with bad export cookie 14252375041400694642 [13314.084572] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13314.094852] LustreError: 338226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13314.545447] Lustre: server umount lustre-MDT0001 complete [13332.014967] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [13335.953975] Key type lgssc unregistered [13336.475688] LNet: 340206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13336.488073] LNetError: 340206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13336.511796] LNet: Removed LNI 192.168.204.128@tcp [13338.131987] Key type .llcrypt unregistered [13338.136096] Key type ._llcrypt unregistered [13366.094679] Key type ._llcrypt registered [13366.097266] Key type .llcrypt registered [13366.296787] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [13395.104854] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [13398.024906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13398.738290] alg: No test for adler32 (adler32-zlib) [13400.938268] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [13401.337451] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [13403.031190] Key type lgssc registered [13405.124902] Lustre: Echo OBD driver; http://www.lustre.org/ [13453.394564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13453.422439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13454.833744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13454.864649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13454.925340] Lustre: lustre-MDT0000: new disk, initializing [13455.010217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13455.029493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13461.735794] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13475.418365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13475.551522] Lustre: 343371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13475.621946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13475.627060] Lustre: Skipped 1 previous similar message [13475.701770] Lustre: lustre-MDT0001: new disk, initializing [13475.842681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13475.890423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13475.903380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13482.730741] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13496.117972] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13504.586977] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13513.274090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13513.678371] Lustre: lustre-OST0000: new disk, initializing [13513.682974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13513.691768] Lustre: 345342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13513.785731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13522.104805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13522.144855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13522.325706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13522.587083] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13533.864967] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13539.406228] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13539.844177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13547.140635] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13547.392713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13553.138983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13553.145626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13553.149237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13553.149773] Lustre: Skipped 1 previous similar message [13553.179891] Lustre: Skipped 1 previous similar message [13555.894835] Lustre: server umount lustre-OST0000 complete [13562.847872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13562.856068] LustreError: Skipped 1 previous similar message [13562.862387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13562.893524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13567.430826] Lustre: server umount lustre-MDT0000 complete [13572.479905] LustreError: 343363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414600 with bad export cookie 9650184111003950868 [13572.482930] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13572.492796] LustreError: 343363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13572.899892] Lustre: server umount lustre-MDT0001 complete [13586.687459] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:16:51 (1786414611) [13611.820352] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [13628.780399] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [13677.579821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13677.807710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13677.838304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13677.895594] Lustre: lustre-MDT0000: new disk, initializing [13677.953794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13677.976632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13684.925524] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13697.235779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13697.378802] Lustre: 350131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13697.494971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13697.497945] Lustre: Skipped 1 previous similar message [13697.711260] Lustre: lustre-MDT0001: new disk, initializing [13697.919274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13697.956490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13697.967357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13704.942206] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13719.326787] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13727.815601] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13738.080977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13738.694760] Lustre: lustre-OST0000: new disk, initializing [13738.708986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13738.734693] Lustre: 352100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13738.924806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13745.846911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13745.871941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13746.137877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13747.568356] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13761.288436] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13812.720560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13812.722158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13812.727796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13812.727807] Lustre: Skipped 2 previous similar messages [13812.743870] Lustre: Skipped 2 previous similar messages [13812.786461] LustreError: Skipped 1 previous similar message [13814.748578] Lustre: server umount lustre-OST0000 complete [13822.947261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13822.949960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13822.967902] Lustre: Skipped 1 previous similar message [13822.990973] Lustre: Skipped 2 previous similar messages [13827.775819] Lustre: server umount lustre-MDT0000 complete [13828.070911] LustreError: 350140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13828.099996] LustreError: 350140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13832.167816] LustreError: 351216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13832.181842] LustreError: 351216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13832.572126] LustreError: 350121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414860 with bad export cookie 9650184111003951750 [13832.575661] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13832.589139] LustreError: 350121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13833.369431] Lustre: server umount lustre-MDT0001 complete [13848.532372] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [13852.389805] Key type lgssc unregistered [13852.803740] LNet: 354280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13852.813597] LNetError: 354280:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13852.823922] LNet: Removed LNI 192.168.204.128@tcp [13854.574981] Key type .llcrypt unregistered [13854.577461] Key type ._llcrypt unregistered [13995.232657] Key type ._llcrypt registered [13995.237419] Key type .llcrypt registered [13995.330722] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [13995.638633] Key type .llcrypt unregistered [13995.640125] Key type ._llcrypt unregistered [14112.868786] Key type ._llcrypt registered [14112.878501] Key type .llcrypt registered [14113.038158] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14113.447377] Key type .llcrypt unregistered [14113.450049] Key type ._llcrypt unregistered [14160.847815] Key type ._llcrypt registered [14160.852136] Key type .llcrypt registered [14161.130217] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14161.592412] Key type .llcrypt unregistered [14161.595168] Key type ._llcrypt unregistered [14207.658323] Key type ._llcrypt registered [14207.666783] Key type .llcrypt registered [14207.898589] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14208.275330] Key type .llcrypt unregistered [14208.279550] Key type ._llcrypt unregistered [14317.470383] Key type ._llcrypt registered [14317.473900] Key type .llcrypt registered [14317.667203] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14318.133780] Key type .llcrypt unregistered [14318.139222] Key type ._llcrypt unregistered [14365.449333] Key type ._llcrypt registered [14365.451823] Key type .llcrypt registered [14365.574217] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14366.167398] Key type .llcrypt unregistered [14366.169178] Key type ._llcrypt unregistered [14410.480939] Key type ._llcrypt registered [14410.484283] Key type .llcrypt registered [14410.675898] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14411.115536] Key type .llcrypt unregistered [14411.120314] Key type ._llcrypt unregistered [14452.644724] Key type ._llcrypt registered [14452.648560] Key type .llcrypt registered [14452.882070] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14453.224847] Key type .llcrypt unregistered [14453.228877] Key type ._llcrypt unregistered [14486.200280] Key type ._llcrypt registered [14486.203741] Key type .llcrypt registered [14486.446451] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14487.084490] Key type .llcrypt unregistered [14487.089443] Key type ._llcrypt unregistered [14525.394514] Key type ._llcrypt registered [14525.396469] Key type .llcrypt registered [14525.521104] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14526.033783] Key type .llcrypt unregistered [14526.037188] Key type ._llcrypt unregistered [14601.278777] Key type ._llcrypt registered [14601.281477] Key type .llcrypt registered [14601.411934] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14601.697137] Key type .llcrypt unregistered [14601.699398] Key type ._llcrypt unregistered [14636.880314] Key type ._llcrypt registered [14636.882366] Key type .llcrypt registered [14637.002835] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14637.305407] Key type .llcrypt unregistered [14637.309306] Key type ._llcrypt unregistered [14675.610465] Key type ._llcrypt registered [14675.614874] Key type .llcrypt registered [14675.727971] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [14676.397470] Key type .llcrypt unregistered [14676.402095] Key type ._llcrypt unregistered [14740.700367] Key type ._llcrypt registered [14740.704886] Key type .llcrypt registered [14740.886925] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [14763.810892] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [14764.889936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14764.956116] alg: No test for adler32 (adler32-zlib) [14766.162349] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [14766.407227] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [14768.087228] Key type lgssc registered [14769.218522] Lustre: Echo OBD driver; http://www.lustre.org/ [14805.869655] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14805.897575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14807.133473] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14807.154995] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14807.217412] Lustre: tst250fs-MDT0000: new disk, initializing [14807.284163] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14807.308171] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14813.423107] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14826.221368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14826.406462] Lustre: 374708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14826.503856] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14826.509183] Lustre: Skipped 1 previous similar message [14826.637883] Lustre: tst250fs-MDT0001: new disk, initializing [14826.784881] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14826.828045] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14826.840661] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14832.605839] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14837.790309] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14845.074980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14845.367538] Lustre: tst250fs-OST0000: new disk, initializing [14845.372348] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14845.379404] Lustre: 376340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14845.504410] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14850.637454] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14850.647866] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14850.880239] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14853.546637] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14866.072640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14866.367323] Lustre: tst250fs-OST0001: new disk, initializing [14866.373968] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14866.384397] Lustre: 377210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14866.559821] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14873.830697] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [14876.901129] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14876.944350] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14877.180550] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14880.262487] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14883.383750] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14883.396967] Lustre: Skipped 2 previous similar messages [14903.777104] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14903.783458] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14903.797925] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14907.882232] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14907.888637] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14907.930558] Lustre: Skipped 2 previous similar messages [14909.249514] Lustre: server umount tst250fs-MDT0000 complete [14912.918125] LustreError: 374698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415940 with bad export cookie 14877214188786151319 [14912.932421] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14912.934919] LustreError: 374698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14913.003363] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14913.017654] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [14913.029296] Lustre: Skipped 1 previous similar message [14913.034141] Lustre: Skipped 3 previous similar messages [14913.334774] Lustre: server umount tst250fs-MDT0001 complete [14923.211766] Lustre: server umount tst250fs-OST0000 complete [14926.766874] Lustre: server umount tst250fs-OST0001 complete [14947.209822] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [14955.046908] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [14987.376484] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:40:13 (1786416013) [14988.972614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15005.866656] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [15021.180686] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [15058.334618] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14753 sec ========= 22:41:23 (1786416083) [15060.071365] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:41:25 (1786416085) === [15063.566346] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:41:29 (1786416089) === [15112.051068] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [15116.119260] Key type lgssc unregistered [15116.564697] LNet: 386378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15116.576904] LNetError: 386378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15117.607347] LNet: Removed LNI 192.168.204.128@tcp [15118.784392] Key type .llcrypt unregistered [15118.793451] Key type ._llcrypt unregistered