[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470212325 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002355] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007442] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010157] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.021834] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026760] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029136] *** VALIDATE tmpfs *** [ 0.031048] *** VALIDATE proc *** [ 0.032143] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041154] debug: unmapping init [mem 0xffffffff86659000-0xffffffff86660fff] [ 0.043911] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052310] rcu: Hierarchical SRCU implementation. [ 0.054527] smp: Bringing up secondary CPUs ... [ 0.055601] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.059415] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.163408] node 0 deferred pages initialised in 101ms [ 0.167169] devtmpfs: initialized [ 0.168263] x86/mm: Memory block size: 128MB [ 0.170936] gcov: version magic: 0x41383552 [ 0.174045] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.176083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.177353] pinctrl core: initialized pinctrl subsystem [ 0.179188] [ 0.179708] ************************************************************* [ 0.182017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184015] ** ** [ 0.187015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189014] ** ** [ 0.192015] ** This means that this kernel is built to expose internal ** [ 0.194012] ** IOMMU data structures, which may compromise security on ** [ 0.196017] ** your system. ** [ 0.198012] ** ** [ 0.201013] ** If you see this message and you are not debugging the ** [ 0.203011] ** kernel, report this immediately to your vendor! ** [ 0.205015] ** ** [ 0.207012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209015] ************************************************************* [ 0.211793] NET: Registered protocol family 16 [ 0.214444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.219066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222173] cpuidle: using governor menu [ 0.223595] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.226506] PCI: Using configuration type 1 for base access [ 0.228194] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237193] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241152] cryptd: max_cpu_qlen set to 1000 [ 0.244361] ACPI: Added _OSI(Module Device) [ 0.246033] ACPI: Added _OSI(Processor Device) [ 0.248022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.249019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261551] ACPI: Interpreter enabled [ 0.262056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264013] ACPI: Using IOAPIC for interrupt routing [ 0.265112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278247] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.283021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.286089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291339] acpiphp: Slot [2] registered [ 0.292117] acpiphp: Slot [5] registered [ 0.294142] acpiphp: Slot [6] registered [ 0.295129] acpiphp: Slot [7] registered [ 0.297127] acpiphp: Slot [8] registered [ 0.298144] acpiphp: Slot [9] registered [ 0.300117] acpiphp: Slot [10] registered [ 0.301112] acpiphp: Slot [3] registered [ 0.303159] acpiphp: Slot [4] registered [ 0.305205] acpiphp: Slot [11] registered [ 0.307155] acpiphp: Slot [12] registered [ 0.308163] acpiphp: Slot [13] registered [ 0.310196] acpiphp: Slot [14] registered [ 0.312166] acpiphp: Slot [15] registered [ 0.313154] acpiphp: Slot [16] registered [ 0.315193] acpiphp: Slot [17] registered [ 0.317185] acpiphp: Slot [18] registered [ 0.319153] acpiphp: Slot [19] registered [ 0.320159] acpiphp: Slot [20] registered [ 0.322185] acpiphp: Slot [21] registered [ 0.323175] acpiphp: Slot [22] registered [ 0.325157] acpiphp: Slot [23] registered [ 0.327223] acpiphp: Slot [24] registered [ 0.328175] acpiphp: Slot [25] registered [ 0.330164] acpiphp: Slot [26] registered [ 0.332180] acpiphp: Slot [27] registered [ 0.333186] acpiphp: Slot [28] registered [ 0.335168] acpiphp: Slot [29] registered [ 0.337141] acpiphp: Slot [30] registered [ 0.338115] acpiphp: Slot [31] registered [ 0.339030] PCI host bridge to bus 0000:00 [ 0.340016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352953] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354957] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367080] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375390] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378370] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383941] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403877] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414558] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421777] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455480] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.511866] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.538020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546854] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.554025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.579021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.589225] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.609027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616037] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.632027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.642000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.654019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.665017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.675609] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.677328] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.679443] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.681000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.684254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.688260] iommu: Default domain type: Passthrough [ 0.689548] SCSI subsystem initialized [ 0.691242] ACPI: bus type USB registered [ 0.693204] usbcore: registered new interface driver usbfs [ 0.695168] usbcore: registered new interface driver hub [ 0.697125] usbcore: registered new device driver usb [ 0.699276] pps_core: LinuxPPS API ver. 1 registered [ 0.701016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.704095] PTP clock support registered [ 0.706092] EDAC MC: Ver: 3.0.0 [ 0.708164] PCI: Using ACPI for IRQ routing [ 0.710370] NetLabel: Initializing [ 0.712017] NetLabel: domain hash size = 128 [ 0.713067] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.715190] NetLabel: unlabeled traffic allowed by default [ 0.718198] vgaarb: loaded [ 0.719435] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.721021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.729000] clocksource: Switched to clocksource kvm-clock [ 0.834671] VFS: Disk quotas dquot_6.6.0 [ 0.836503] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.839229] *** VALIDATE ramfs *** [ 0.840636] *** VALIDATE hugetlbfs *** [ 0.842517] pnp: PnP ACPI init [ 0.844871] pnp: PnP ACPI: found 6 devices [ 0.862209] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866040] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.867945] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.869657] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.871759] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874178] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.876453] NET: Registered protocol family 2 [ 0.878476] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.882222] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.885481] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.890070] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.893074] TCP: Hash tables configured (established 65536 bind 65536) [ 0.895841] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.899432] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.902305] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.905628] NET: Registered protocol family 1 [ 0.908632] RPC: Registered named UNIX socket transport module. [ 0.911393] RPC: Registered udp transport module. [ 0.913233] RPC: Registered tcp transport module. [ 0.915209] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.917685] NET: Registered protocol family 44 [ 0.919328] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.921036] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.923063] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.925195] PCI: CLS 0 bytes, default 64 [ 0.926782] Unpacking initramfs... [ 2.355233] debug: unmapping init [mem 0xffff9ee07cc54000-0xffff9ee07ffbffff] [ 2.359098] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.361280] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.364063] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.856956] Initialise system trusted keyrings [ 2.858890] Key type blacklist registered [ 2.860936] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.870046] zbud: loaded [ 2.872954] *** VALIDATE nfs *** [ 2.874180] *** VALIDATE nfs4 *** [ 2.875663] pstore: using deflate compression [ 2.878143] Platform Keyring initialized [ 2.977597] NET: Registered protocol family 38 [ 2.979669] Key type asymmetric registered [ 2.980870] Asymmetric key parser 'x509' registered [ 2.982287] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.984360] io scheduler mq-deadline registered [ 2.985937] io scheduler kyber registered [ 2.988162] io scheduler bfq registered [ 2.990936] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.993854] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.995865] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.997699] ACPI: Power Button [PWRF] [ 3.002200] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.006920] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.014833] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.020485] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.034102] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.062937] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.092854] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.096958] Non-volatile memory driver v1.3 [ 3.098254] Linux agpgart interface v0.103 [ 3.126253] virtio_blk virtio1: [vda] 147040 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.128093] vda: detected capacity change from 0 to 75284480 [ 3.139325] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.142753] vdb: detected capacity change from 0 to 1073741824 [ 3.160536] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.162585] vdc: detected capacity change from 0 to 2621440000 [ 3.179217] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.181049] vdd: detected capacity change from 0 to 2621440000 [ 3.197057] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.199954] vde: detected capacity change from 0 to 4294967296 [ 3.211860] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.214183] vdf: detected capacity change from 0 to 4294967296 [ 3.219986] libphy: Fixed MDIO Bus: probed [ 3.224163] usbcore: registered new interface driver usbserial_generic [ 3.225826] usbserial: USB Serial support registered for generic [ 3.227459] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.230320] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.231384] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.232949] mousedev: PS/2 mouse device common for all mice [ 3.234881] rtc_cmos 00:05: RTC can wake from S4 [ 3.236630] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.238735] rtc_cmos 00:05: registered as rtc0 [ 3.243550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.243589] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.248522] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.249212] intel_pstate: CPU model not supported [ 3.253479] hid: raw HID events driver (C) Jiri Kosina [ 3.254821] usbcore: registered new interface driver usbhid [ 3.256332] usbhid: USB HID core driver [ 3.257410] drop_monitor: Initializing network drop monitor service [ 3.258942] Initializing XFRM netlink socket [ 3.260177] NET: Registered protocol family 10 [ 3.262231] Segment Routing with IPv6 [ 3.263331] NET: Registered protocol family 17 [ 3.265220] mpls_gso: MPLS GSO support [ 3.269676] RAS: Correctable Errors collector initialized. [ 3.271031] AVX version of gcm_enc/dec engaged. [ 3.272188] AES CTR mode by8 optimization enabled [ 3.328799] sched_clock: Marking stable (3328777617, 0)->(4259933777, -931156160) [ 3.331856] registered taskstats version 1 [ 3.333292] Loading compiled-in X.509 certificates [ 3.335281] zswap: loaded using pool lzo/zbud [ 3.354265] Key type big_key registered [ 3.362946] Key type encrypted registered [ 3.363960] ima: No TPM chip found, activating TPM-bypass! [ 3.365331] ima: Allocated hash algorithm: sha1 [ 3.366462] ima: No architecture policies found [ 3.367545] evm: Initialising EVM extended attributes: [ 3.368729] evm: security.selinux [ 3.369390] evm: security.ima [ 3.370112] evm: security.capability [ 3.370890] evm: HMAC attrs: 0x1 [ 3.372511] rtc_cmos 00:05: setting system clock to 2026-08-07 03:49:23 UTC (1786074563) [ 3.377374] debug: unmapping init [mem 0xffffffff87603000-0xffffffff877fffff] [ 3.379264] debug: unmapping init [mem 0xffffffff86382000-0xffffffff86658fff] [ 3.386148] Write protecting the kernel read-only data: 28672k [ 3.389484] debug: unmapping init [mem 0xffffffff84a03000-0xffffffff84bfffff] [ 3.392232] debug: unmapping init [mem 0xffffffff85314000-0xffffffff853fffff] [ 3.425412] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.433335] systemd[1]: Detected virtualization kvm. [ 3.435130] systemd[1]: Detected architecture x86-64. [ 3.437046] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.463423] systemd[1]: No hostname configured. [ 3.464677] systemd[1]: Set hostname to . [ 3.466124] random: systemd: uninitialized urandom read (16 bytes read) [ 3.467474] systemd[1]: Initializing machine ID from random generator. [ 3.512736] random: ln: uninitialized urandom read (6 bytes read) [ 3.588481] random: systemd: uninitialized urandom read (16 bytes read) [ 3.591124] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.596178] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.600591] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.138821] device-mapper: uevent: version 1.0.3 [ 4.141053] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Col[ 4.744556] random: fast init done dplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.817398] virtio_net virtio0 ens2: renamed from eth0 [ 4.848287] scsi host0: ata_piix [ 4.853637] scsi host1: ata_piix [ 4.855593] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.859182] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.073257] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.879125] random: crng init done [ 9.880371] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.121388] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.201614] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.439625] SELinux: Disabled at runtime. [ 11.499364] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.507258] systemd[1]: Detected virtualization kvm. [ 11.509036] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.977631] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.983429] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.990267] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.994762] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.997926] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.003815] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.007147] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.137172] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt.[ 12.418986] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.725805] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.748265] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.853783] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.872179] EDAC sbridge: Ver: 1.1.2 [ 14.428304] Key type dns_resolver registered [ 14.740688] NFS: Registering the id_resolver key type [ 14.742830] Key type id_resolver registered [ 14.744555] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-server login: [ 53.228851] libcfs: loading out-of-tree module taints kernel. [ 53.361520] Key type ._llcrypt registered [ 53.366745] Key type .llcrypt registered [ 53.471952] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 75.271356] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 77.328151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.354790] alg: No test for adler32 (adler32-zlib) [ 78.929784] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 79.861883] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 81.607652] Key type lgssc registered [ 83.501582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.748710] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.684160] hrtimer: interrupt took 3761559 ns [ 145.696240] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 161.935719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.025558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.271109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.322121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.505298] Lustre: lustre-MDT0000: new disk, initializing [ 163.615938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.662290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.970459] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.892839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.129639] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.164132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.169184] Lustre: Skipped 1 previous similar message [ 184.339877] Lustre: lustre-MDT0001: new disk, initializing [ 184.475190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.528484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.536709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.848920] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.617304] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.768834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.984995] Lustre: lustre-OST0000: new disk, initializing [ 204.988608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.993827] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.058156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.039706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 210.066178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.194252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.237518] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.254162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.421726] Lustre: lustre-OST0001: new disk, initializing [ 228.427915] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.433593] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.511737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.968923] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.134381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.153715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.291549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.779596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.388831] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 263.756851] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing check_logdir /tmp/testlogs/ [ 269.972369] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing yml_node [ 275.279993] Lustre: DEBUG MARKER: Client: 2.17.56.3 [ 278.122203] Lustre: DEBUG MARKER: MDS: 2.17.56.3 [ 281.236456] Lustre: DEBUG MARKER: OSS: 2.17.56.3 [ 283.116475] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 23:54:01 EDT 2026 [ 301.923819] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 303.644824] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.483936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.494780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.499790] Lustre: Skipped 3 previous similar messages [ 320.508531] Lustre: Skipped 3 previous similar messages [ 325.599851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.610732] Lustre: Skipped 3 previous similar messages [ 326.912366] Lustre: server umount lustre-MDT0000 complete [ 330.916414] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786074891 with bad export cookie 14813951166605617731 [ 330.924232] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.926532] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 335.841101] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.843334] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.887271] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.939783] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 337.467116] Lustre: server umount lustre-MDT0001 complete [ 342.285772] Lustre: server umount lustre-OST0000 complete [ 346.828914] Lustre: server umount lustre-OST0001 complete [ 355.547663] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 367.633725] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 413.168179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.472366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.533752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.628322] Lustre: lustre-MDT0000: new disk, initializing [ 413.742799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.787225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 419.315148] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 433.340436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.482183] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.573773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.583316] Lustre: Skipped 1 previous similar message [ 433.686496] Lustre: lustre-MDT0001: new disk, initializing [ 433.808738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.878609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.897736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.252952] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 452.338847] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 459.903344] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.649276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 468.978824] Lustre: lustre-OST0000: new disk, initializing [ 468.985368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 469.003820] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 469.106614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.415148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.428905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.553786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 476.633732] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 488.859265] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 494.331824] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.637626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 500.190822] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 500.548140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 506.341067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.352651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.375769] Lustre: Skipped 2 previous similar messages [ 506.407251] Lustre: Skipped 2 previous similar messages [ 509.214823] Lustre: server umount lustre-OST0000 complete [ 516.065262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 516.077441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.100753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.935911] Lustre: server umount lustre-MDT0000 complete [ 521.696900] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.710097] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 524.126927] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075084 with bad export cookie 14813951166605619264 [ 524.140292] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.142394] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 524.550767] Lustre: server umount lustre-MDT0001 complete [ 531.918332] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:58:09 (1786075089) [ 555.175874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.635859] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.752766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.131822] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.356722] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 565.226175] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.459723] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.495557] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 576.250450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.824361] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 592.323869] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.463891] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.159927] Lustre: Failing over lustre-MDT0000 [ 602.592081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 602.599188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.629224] Lustre: Skipped 1 previous similar message [ 602.644414] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.715728] Lustre: server umount lustre-MDT0000 complete [ 623.583316] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786075167/real 1786075167] req@ffff9edfc79cf480 x1872835000832384/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786075183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 623.596402] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.807531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.827149] LustreError: 3641:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ee0ff117800 x1872835000834816/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 634.240231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.246358] Lustre: Skipped 1 previous similar message [ 634.311851] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 639.457606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 639.474869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 639.498349] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 639.858180] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 651.250188] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 656.867222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.878059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 656.878679] Lustre: Skipped 2 previous similar messages [ 656.897158] Lustre: Skipped 2 previous similar messages [ 670.175243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 670.612525] Lustre: server umount lustre-MDT0000 complete [ 671.203631] LustreError: 22149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.224738] LustreError: 22149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 675.696795] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075235 with bad export cookie 14813951166605620678 [ 675.738067] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.997849] Lustre: server umount lustre-MDT0001 complete [ 689.032170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.161141] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 706.242553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.488384] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 722.317103] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 729.279522] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 731.847341] Lustre: Failing over lustre-MDT0000 [ 732.130435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 732.142357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.249761] Lustre: server umount lustre-MDT0000 complete [ 736.224680] LustreError: 23265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.260118] LustreError: 23265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 752.615640] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786075296/real 1786075296] req@ffff9edfc79c0a80 x1872835000864384/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786075312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 752.668170] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 752.710805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.385148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 763.396903] Lustre: Skipped 2 previous similar messages [ 763.481637] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 768.491701] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 768.496635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 768.515201] Lustre: Skipped 1 previous similar message [ 768.558036] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 768.891742] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 779.896369] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 786.921502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.923029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.932967] Lustre: Skipped 2 previous similar messages [ 786.952676] Lustre: Skipped 5 previous similar messages [ 791.471193] Lustre: server umount lustre-MDT0000 complete [ 796.010892] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075356 with bad export cookie 14813951166605621616 [ 796.030257] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 796.038491] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 796.613555] Lustre: server umount lustre-MDT0001 complete [ 806.778507] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:02:44 (1786075364) [ 828.368271] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 840.605602] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 889.588037] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 902.815573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.222354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 903.250943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 903.354670] Lustre: lustre-MDT0000: new disk, initializing [ 903.500554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 910.088756] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 923.570518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.722943] Lustre: 29656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 923.934377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 929.714469] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 941.379950] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 947.306974] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 954.749965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.014618] Lustre: 31628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 957.092830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 957.132824] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 957.235741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 957.245966] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 961.980440] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 976.706384] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 985.993020] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1008.610641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1008.622354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.637349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1008.647727] Lustre: Skipped 1 previous similar message [ 1013.109517] Lustre: server umount lustre-OST0000 complete [ 1025.388555] Lustre: server umount lustre-MDT0000 complete [ 1028.065118] LustreError: 30528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.078088] LustreError: 30528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 1029.681460] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075589 with bad export cookie 14813951166605622022 [ 1029.688506] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1029.691651] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1030.241373] Lustre: server umount lustre-MDT0001 complete [ 1044.468622] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1048.497802] Key type lgssc unregistered [ 1049.004716] LNet: 33608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.042263] LNetError: 33608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.063972] LNet: Removed LNI 192.168.201.106@tcp [ 1050.468162] Key type .llcrypt unregistered [ 1050.469572] Key type ._llcrypt unregistered [ 1082.045987] Key type ._llcrypt registered [ 1082.050868] Key type .llcrypt registered [ 1082.260977] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1084.036087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1084.162350] alg: No test for adler32 (adler32-zlib) [ 1085.434716] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1085.675526] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1087.379419] Key type lgssc registered [ 1088.848614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1098.003074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1098.026371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.705397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.438983] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1109.027225] LustreError: 35025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.043331] LustreError: 35025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1114.087159] LustreError: 35024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.949838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.637555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1123.213333] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1133.334948] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1141.391511] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.608846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.196384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.143943] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1162.493303] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1168.521123] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1191.297447] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:09:08 (1786075748) [ 1193.801906] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1195.991717] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1205.141431] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:09:22 (1786075762) [ 1215.801247] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:09:33 (1786075773) [ 1217.278866] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1228.062497] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1228.076900] Lustre: Skipped 2 previous similar messages [ 1229.932860] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:09:48 (1786075788) [ 1234.174986] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1234.180942] Lustre: Skipped 3 previous similar messages [ 1254.178644] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1254.193300] Lustre: Skipped 6 previous similar messages [ 1257.938435] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:10:15 (1786075815) [ 1261.368771] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1266.089229] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1276.709938] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:10:34 (1786075834) [ 1279.678665] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1279.682125] Lustre: Skipped 3 previous similar messages [ 1294.774554] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:10:52 (1786075852) [ 1311.203201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1311.215289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.228444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1313.258818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.259750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1313.260027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.311162] Lustre: Skipped 1 previous similar message [ 1316.920680] Lustre: server umount lustre-OST0000 complete [ 1321.440136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1321.444418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.454657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1323.489680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1323.503250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.392516] Lustre: server umount lustre-MDT0000 complete [ 1328.608645] LustreError: 35024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.629230] LustreError: 35024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1332.182514] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075892 with bad export cookie 4897291943392724590 [ 1332.183551] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.201249] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1332.906963] Lustre: server umount lustre-MDT0001 complete [ 1346.680843] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1350.295541] Key type lgssc unregistered [ 1350.709568] LNet: 42883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.738670] LNetError: 42883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.781962] LNet: Removed LNI 192.168.201.106@tcp [ 1352.772815] Key type .llcrypt unregistered [ 1352.776757] Key type ._llcrypt unregistered [ 1374.248974] Key type ._llcrypt registered [ 1374.251432] Key type .llcrypt registered [ 1374.369879] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 1391.084927] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1392.285907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1392.405062] alg: No test for adler32 (adler32-zlib) [ 1393.747562] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1394.031072] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1395.703331] Key type lgssc registered [ 1397.253890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.193617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.267483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.672426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1445.738771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1445.879496] Lustre: lustre-MDT0000: new disk, initializing [ 1446.038773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.076592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1452.000948] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1464.739786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.909499] Lustre: 46044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1464.976563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1464.981372] Lustre: Skipped 1 previous similar message [ 1465.103628] Lustre: lustre-MDT0001: new disk, initializing [ 1465.293429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1465.353577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1465.375804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1471.809797] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1482.657593] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1490.163370] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1498.972911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1499.476416] Lustre: lustre-OST0000: new disk, initializing [ 1499.482608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1499.495253] Lustre: 48012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1499.633149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.451380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1507.468481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1507.666320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1509.009798] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1521.408822] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1527.678968] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1527.935100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1533.886447] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1534.137176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1538.528216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1538.531157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.544433] LustreError: Skipped 1 previous similar message [ 1538.555195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.619982] Lustre: Skipped 1 previous similar message [ 1542.802387] Lustre: server umount lustre-OST0000 complete [ 1547.235900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1547.248512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.266493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.270698] Lustre: Skipped 1 previous similar message [ 1548.780905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.788504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1548.805856] Lustre: Skipped 1 previous similar message [ 1553.265490] Lustre: server umount lustre-MDT0000 complete [ 1557.581878] LustreError: 46036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076117 with bad export cookie 42479774310071178 [ 1557.590468] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1558.265296] Lustre: server umount lustre-MDT0001 complete [ 1570.799558] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:15:28 (1786076128) [ 1578.359653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.012976] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.034035] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1579.132340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.612201] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.438645] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1589.730831] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.852550] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.597188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.297130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1602.738976] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1612.595534] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1620.455764] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1629.844619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.333264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.788660] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1650.067585] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1655.941109] LustreError: 53005:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1657.228887] Lustre: Modifying parameter general.timeout=1 in log params [ 1658.711660] Lustre: Modifying parameter general.timeout=2 in log params [ 1661.734950] Lustre: Modifying parameter general.timeout=4 in log params [ 1661.740978] Lustre: Skipped 1 previous similar message [ 1666.026967] Lustre: Modifying parameter general.timeout=7 in log params [ 1666.030228] Lustre: Skipped 2 previous similar messages [ 1674.234312] Lustre: Modifying parameter general.timeout=13 in log params [ 1674.239396] Lustre: Skipped 5 previous similar messages [ 1690.402670] Lustre: Modifying parameter general.timeout=25 in log params [ 1690.407320] Lustre: Skipped 11 previous similar messages [ 1723.535782] Lustre: Modifying parameter general.timeout=55 in log params [ 1723.539712] Lustre: Skipped 29 previous similar messages [ 1788.789380] Lustre: Modifying parameter general.timeout=110 in log params [ 1788.792036] Lustre: Skipped 54 previous similar messages [ 1828.868257] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:19:46 (1786076386) [ 1832.361569] Lustre: 59971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1834.612560] Lustre: 60069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1836.903346] Lustre: 60168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1841.515155] Lustre: 60384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1841.540247] Lustre: 60384:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1879.203728] Lustre: 61858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1879.217358] Lustre: 61858:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1879.263535] LustreError: 61858:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1885.112661] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:20:42 (1786076442) [ 1904.612059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.618410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1904.640138] Lustre: Skipped 1 previous similar message [ 1904.661850] Lustre: Skipped 2 previous similar messages [ 1909.739300] LustreError: 50469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1909.771683] LustreError: 50469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1909.859336] Lustre: server umount lustre-MDT0000 complete [ 1914.592338] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076474 with bad export cookie 42479774310072060 [ 1914.601767] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1914.611863] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1914.848916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.866546] Lustre: Skipped 1 previous similar message [ 1915.046083] Lustre: server umount lustre-MDT0001 complete [ 1920.714844] Lustre: server umount lustre-OST0000 complete [ 1955.673725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.019955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1956.089370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1956.099564] Lustre: Skipped 42 previous similar messages [ 1956.527318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.271715] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1973.526416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.617526] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1973.637431] Lustre: 64392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1982.586275] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1996.893869] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2006.940681] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2017.921218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.381788] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2019.012244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.018706] Lustre: Skipped 1 previous similar message [ 2026.643133] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2036.418485] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2038.655678] Lustre: 66960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2050.529795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2050.535789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.569283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2053.525733] Lustre: server umount lustre-OST0000 complete [ 2060.768168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2060.797507] LustreError: Skipped 1 previous similar message [ 2060.808383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.816260] Lustre: Skipped 2 previous similar messages [ 2064.977902] Lustre: server umount lustre-MDT0000 complete [ 2066.405922] LustreError: 64402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.430244] LustreError: 64402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2070.244648] LustreError: 64384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076630 with bad export cookie 42479774310075455 [ 2070.260154] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.083154] Lustre: server umount lustre-MDT0001 complete [ 2086.087610] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2090.650851] Key type lgssc unregistered [ 2091.137746] LNet: 68170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.145893] LNetError: 68170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.173859] LNet: Removed LNI 192.168.201.106@tcp [ 2092.500608] Key type .llcrypt unregistered [ 2092.503690] Key type ._llcrypt unregistered [ 2127.291291] Key type ._llcrypt registered [ 2127.294902] Key type .llcrypt registered [ 2127.482893] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2129.361697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.607128] alg: No test for adler32 (adler32-zlib) [ 2131.065975] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2131.451784] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2133.383288] Key type lgssc registered [ 2135.821210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2145.196499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2145.213923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.047043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.199080] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2157.538621] LustreError: 69586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.562134] LustreError: 69586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2162.657089] LustreError: 69587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.824683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.714567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2173.338608] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2183.436591] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2191.464826] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2201.432937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.045140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.481679] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2220.643181] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2230.739635] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2244.066058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2244.089376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.119808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.168364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.170555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.204631] Lustre: Skipped 1 previous similar message [ 2248.227337] Lustre: Skipped 2 previous similar messages [ 2249.919608] Lustre: server umount lustre-MDT0000 complete [ 2253.286995] LustreError: 70982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.314685] LustreError: 70982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2254.984463] LustreError: 69567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076815 with bad export cookie 10155992171315319362 [ 2254.996196] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.584136] Lustre: server umount lustre-MDT0001 complete [ 2271.588179] Lustre: server umount lustre-OST0000 complete [ 2309.188870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.740163] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2309.825166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2310.289035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.204565] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2328.805427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.922675] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2328.970253] Lustre: 74461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2329.789247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2335.899599] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2346.686570] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2354.873566] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2363.774866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2364.014147] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2364.460157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.300311] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2380.769902] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2382.460409] Lustre: 77042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2384.754361] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2394.085095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.092235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.117729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.703890] Lustre: server umount lustre-OST0000 complete [ 2404.833505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.874653] Lustre: Skipped 1 previous similar message [ 2404.886876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.889400] Lustre: Skipped 1 previous similar message [ 2408.367541] Lustre: server umount lustre-MDT0000 complete [ 2409.954664] LustreError: 74471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.989414] LustreError: 74471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2413.094174] LustreError: 75209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076973 with bad export cookie 10155992171315320209 [ 2413.101042] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2413.108036] LustreError: 75209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2413.595251] Lustre: server umount lustre-MDT0001 complete [ 2427.180542] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2431.508959] Key type lgssc unregistered [ 2431.768757] LNet: 78256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.775811] LNetError: 78256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.789155] LNet: Removed LNI 192.168.201.106@tcp [ 2432.623250] Key type .llcrypt unregistered [ 2432.626523] Key type ._llcrypt unregistered [ 2463.602632] Key type ._llcrypt registered [ 2463.607377] Key type .llcrypt registered [ 2463.766693] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2465.592188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.630019] alg: No test for adler32 (adler32-zlib) [ 2466.896843] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2467.223929] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2468.927308] Key type lgssc registered [ 2470.199300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2479.720535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2479.738261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.426895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.387189] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2491.877944] LustreError: 79669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.905785] LustreError: 79669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2496.631155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.129451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2502.674980] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2514.364754] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2523.035307] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2530.057176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2530.669464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.665190] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2547.025484] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2560.912860] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2562.479801] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2564.110471] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2567.009315] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2567.013423] Lustre: Skipped 1 previous similar message [ 2571.613271] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2571.620409] Lustre: Skipped 2 previous similar messages [ 2579.746063] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2579.748729] Lustre: Skipped 6 previous similar messages [ 2597.396271] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2597.404915] Lustre: Skipped 14 previous similar messages [ 2630.379817] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2630.395071] Lustre: Skipped 27 previous similar messages [ 2684.285937] Lustre: 87815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2687.435530] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:34:05 (1786077245) [ 2689.481800] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2691.821363] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:34:09 (1786077249) [ 2699.758063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.763253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.771109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.771157] Lustre: Skipped 1 previous similar message [ 2703.516867] Lustre: server umount lustre-OST0000 complete [ 2709.988405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2710.005107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2710.011490] Lustre: Skipped 1 previous similar message [ 2710.037885] Lustre: Skipped 2 previous similar messages [ 2714.975501] Lustre: server umount lustre-MDT0000 complete [ 2715.108218] LustreError: 79669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.116733] LustreError: 79669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2719.653547] LustreError: 79652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786077279 with bad export cookie 9547155928128299320 [ 2719.659537] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.666261] LustreError: 79652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2719.995482] Lustre: server umount lustre-MDT0001 complete [ 2732.587634] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2735.492322] Key type lgssc unregistered [ 2735.872125] LNet: 89065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.890515] LNetError: 89065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2735.924825] LNet: Removed LNI 192.168.201.106@tcp [ 2736.886284] Key type .llcrypt unregistered [ 2736.888958] Key type ._llcrypt unregistered [ 2742.221062] Key type ._llcrypt registered [ 2742.222152] Key type .llcrypt registered [ 2742.367461] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2747.334741] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules [ 2748.460485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2748.558395] alg: No test for adler32 (adler32-zlib) [ 2749.778332] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2750.118793] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2751.855232] Key type lgssc registered [ 2753.227169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2754.546610] LustreError: 90274:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2773.998589] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2776.610586] LustreError: 90274:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2784.270995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2784.288592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.839566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.924877] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2794.982402] LustreError: 91299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.010528] LustreError: 91299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2800.097601] LustreError: 91300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.344677] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:35:58 (1786077358) [ 2805.222909] LustreError: 91299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.906779] LustreError: 91331:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2805.923327] LustreError: 91331:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2805.939797] LustreError: 91331:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2806.314935] Lustre: server umount lustre-MDT0000 complete [ 2822.221931] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2825.480343] Key type lgssc unregistered [ 2825.810509] LNet: 92825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2825.830171] LNetError: 92825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2825.857365] LNet: Removed LNI 192.168.201.106@tcp [ 2827.012716] Key type .llcrypt unregistered [ 2827.015424] Key type ._llcrypt unregistered [ 2849.110840] Key type ._llcrypt registered [ 2849.113528] Key type .llcrypt registered [ 2849.176252] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2850.744179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2850.759781] alg: No test for adler32 (adler32-zlib) [ 2851.909937] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2852.180218] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2853.928050] Key type lgssc registered [ 2855.249587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2864.496749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2864.534169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.291171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.377941] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2876.392302] LustreError: 94045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.416400] LustreError: 94045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2881.516686] LustreError: 94046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.627937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.671830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.932710] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2897.937214] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2905.561597] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2913.263866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.928593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.940748] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2929.911908] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3001.211875] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:39:18 (1786077558) [ 3003.407971] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3005.739680] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:39:23 (1786077563) [ 3013.603736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.607105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.633600] Lustre: Skipped 2 previous similar messages [ 3013.650578] Lustre: Skipped 2 previous similar messages [ 3017.674741] Lustre: server umount lustre-MDT0000 complete [ 3018.724363] LustreError: 94045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.751579] LustreError: 94045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3021.329355] LustreError: 94026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786077581 with bad export cookie 7231782397829936703 [ 3021.356933] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3021.820378] Lustre: server umount lustre-MDT0001 complete [ 3036.893366] Lustre: server umount lustre-OST0000 complete [ 3053.005685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.890294] LustreError: 98366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.222666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.554958] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3059.688421] LustreError: 98367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.703743] LustreError: 98367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3067.310918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.248077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3073.210835] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3080.886368] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3086.772467] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3101.958948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.277547] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3102.291532] LustreError: 98353:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3102.310859] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3102.316730] LustreError: 100450:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3102.321876] LustreError: 100450:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3102.329987] LustreError: 100450:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3102.456365] Lustre: server umount lustre-OST0000 complete [ 3102.458219] LustreError: 100450:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3112.883526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.142198] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3113.146517] LustreError: 98356:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3113.175112] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3113.183811] LustreError: 100916:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3113.193144] LustreError: 100916:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3113.199601] LustreError: 100916:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3113.307416] Lustre: server umount lustre-OST0000 complete [ 3113.314179] LustreError: 100916:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3125.099634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3125.281395] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3125.286922] Lustre: Found index 0 for lustre-OST0000, updating log [ 3125.293309] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3125.373483] Lustre: lustre-OST0000: new disk, initializing [ 3125.378344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3125.387430] Lustre: 101452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3125.664551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.801315] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3136.916432] Lustre: Failing over lustre-OST0000 [ 3137.071501] Lustre: server umount lustre-OST0000 complete [ 3143.138498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.159252] Lustre: Skipped 1 previous similar message [ 3143.165387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.179524] Lustre: Skipped 1 previous similar message [ 3147.840491] Lustre: server umount lustre-MDT0000 complete [ 3148.257975] LustreError: 98362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.273131] LustreError: 98362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3152.323180] LustreError: 98347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786077712 with bad export cookie 7231782397829939440 [ 3152.328472] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.332708] LustreError: 98347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3152.883264] Lustre: server umount lustre-MDT0001 complete [ 3162.520439] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:42:00 (1786077720) [ 3183.875297] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3196.943759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.387328] LustreError: 103800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.584691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.465594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3213.255102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.294756] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3225.025259] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3232.353596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3232.920480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.934785] Lustre: Skipped 1 previous similar message [ 3241.373677] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3251.185103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3251.242102] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3251.287340] Lustre: lustre-OST0001: new disk, initializing [ 3251.289128] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3251.291848] Lustre: 106119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3257.479048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3257.501156] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3257.630756] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3258.134207] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3268.179402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3276.620361] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3280.864377] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3280.869200] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.882193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3283.440593] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3283.450306] Lustre: Skipped 1 previous similar message [ 3283.455608] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.482636] Lustre: Skipped 1 previous similar message [ 3288.555961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3288.564216] Lustre: Skipped 5 previous similar messages [ 3295.199574] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3295.386375] Lustre: server umount lustre-MDT0001 complete [ 3297.391441] LustreError: 105523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.422837] LustreError: 105523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3307.257450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.438722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3307.447361] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3307.526600] Lustre: 103789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3308.444514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3308.452885] Lustre: Skipped 1 previous similar message [ 3308.709921] LustreError: 107646:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3308.720187] LustreError: 107646:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3308.738193] LustreError: 103793:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3308.974089] LustreError: 107663:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.106@tcp: cfg command failed: rc = -17 [ 3308.988604] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3308.988604] [ 3313.649461] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3314.278600] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3326.572434] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:44:44 (1786077884) [ 3347.008441] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3366.351627] Lustre: 109748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3391.250873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3394.632655] Lustre: 110883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3492.320182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3492.328673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.348944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3492.359984] Lustre: Skipped 4 previous similar messages [ 3494.929954] Lustre: server umount lustre-MDT0000 complete [ 3497.952371] LustreError: 103796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.981340] LustreError: 103796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3499.064245] LustreError: 103782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078059 with bad export cookie 7231782397829940441 [ 3499.065190] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3499.071023] LustreError: 103782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3499.414886] Lustre: server umount lustre-MDT0001 complete [ 3514.682453] Lustre: server umount lustre-OST0000 complete [ 3529.833570] Lustre: server umount lustre-OST0001 complete [ 3536.588814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3548.119948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3557.151965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3568.131969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3589.616241] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3601.298451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.316497] Lustre: lustre-MDT0000: reset Object Index mappings [ 3602.249344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.449971] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3617.578501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.619552] Lustre: lustre-MDT0001: reset Object Index mappings [ 3617.759993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3617.765613] Lustre: Skipped 3 previous similar messages [ 3622.869372] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3626.912183] Lustre: 115994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3636.129981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3642.470982] LustreError: 116346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.487760] LustreError: 116346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3642.495353] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3645.189255] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3647.638590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3654.135784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.773397] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3659.792977] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3661.296939] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3669.628417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3673.760860] Lustre: 117869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3722.822260] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:51:21 (1786078281) [ 3750.882298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3750.892604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.912066] Lustre: Skipped 3 previous similar messages [ 3750.919655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.903969] Lustre: server umount lustre-MDT0000 complete [ 3761.458850] LustreError: 114826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078321 with bad export cookie 7231782397829987313 [ 3761.464422] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3761.476982] LustreError: 114826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3768.218286] Lustre: server umount lustre-MDT0001 complete [ 3772.839386] LustreError: 118628:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0xb7:0x0].0x0 (ffff9edfcb969d00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3787.232468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3787.402256] Lustre: server umount lustre-OST0000 complete [ 3791.498656] LustreError: 118832:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x3f:0x0].0x0 (ffff9ee0d01c1300) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3797.988473] Lustre: server umount lustre-OST0001 complete [ 3806.924351] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 3819.763442] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3861.493798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.765514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3861.794480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3861.919367] Lustre: lustre-MDT0000: new disk, initializing [ 3862.061376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.065658] Lustre: Skipped 3 previous similar messages [ 3862.089564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3866.850808] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3874.022762] Lustre: server umount lustre-MDT0000 complete [ 3882.931550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.165983] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3883.182961] Lustre: 122144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3883.197263] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3888.015546] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3893.173339] Lustre: server umount lustre-MDT0000 complete [ 3903.179411] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:54:21 (1786078461) [ 3904.601840] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3906.213033] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:54:24 (1786078464) [ 3922.966953] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3935.950405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.778900] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3953.619167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.822691] Lustre: 124191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3953.999768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3954.002490] Lustre: Skipped 1 previous similar message [ 3954.188637] Lustre: lustre-MDT0001: new disk, initializing [ 3954.437361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3954.456940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3959.373298] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3964.323955] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3972.232643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.622195] Lustre: lustre-OST0000: new disk, initializing [ 3972.625777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3972.633692] Lustre: 125772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3973.995667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3974.017976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3974.260756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.032301] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3994.129773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.334472] Lustre: lustre-OST0001: new disk, initializing [ 3994.337986] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3994.343158] Lustre: 126644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3996.477231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3996.489493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3996.653196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4001.270249] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4010.476465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4014.586209] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4459.312336] Lustre: 133086:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4459.333815] Lustre: 133086:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4459.348490] Lustre: 133086:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4459.362838] Lustre: 133086:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4459.369579] Lustre: 133086:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4459.383461] Lustre: 133086:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4786.588820] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:09:04 (1786079344) [ 4804.582663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4804.584555] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.587172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4804.587182] Lustre: Skipped 8 previous similar messages [ 4804.633356] Lustre: Skipped 8 previous similar messages [ 4809.684197] Lustre: server umount lustre-MDT0000 complete [ 4809.730310] LustreError: 127462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4809.762201] LustreError: 127462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 4814.872666] LustreError: 124183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786079374 with bad export cookie 7231782397830055675 [ 4814.884669] LustreError: 124183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4814.885575] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.943511] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.964204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4819.973069] Lustre: Skipped 4 previous similar messages [ 4821.500469] Lustre: server umount lustre-MDT0001 complete [ 4832.088577] Lustre: server umount lustre-OST0000 complete [ 4851.167488] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4851.384438] Lustre: server umount lustre-OST0001 complete [ 4858.972454] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 4871.022613] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4909.979688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4910.190996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4910.210678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4910.279919] Lustre: lustre-MDT0000: new disk, initializing [ 4910.344334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.347654] Lustre: Skipped 5 previous similar messages [ 4910.359468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4914.516695] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4925.592235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.690103] Lustre: 136147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4925.746075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4925.748822] Lustre: Skipped 1 previous similar message [ 4925.820932] Lustre: lustre-MDT0001: new disk, initializing [ 4925.931249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4925.942056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4931.097576] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4941.640334] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4948.879869] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4955.782717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4956.032751] Lustre: lustre-OST0000: new disk, initializing [ 4956.035757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4956.048139] Lustre: 138115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4957.369456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4957.375831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4957.499403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4964.847963] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4977.304525] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4988.396495] Lustre: lustre-MDD0000: changelog on [ 4991.661765] Lustre: lustre-MDD0001: changelog on [ 7029.041819] Lustre: lustre-MDD0001: changelog off [ 7031.233266] Lustre: 136154:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7032.215617] Lustre: lustre-MDD0000: changelog off [ 7041.514696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7041.515418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.525849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7041.537326] LustreError: Skipped 1 previous similar message [ 7041.542406] Lustre: Skipped 2 previous similar messages [ 7046.627103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.648131] Lustre: Skipped 2 previous similar messages [ 7051.750509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7051.758942] Lustre: Skipped 1 previous similar message [ 7052.255355] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7052.467387] Lustre: server umount lustre-OST0000 complete [ 7056.867026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7061.985471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.995759] Lustre: Skipped 3 previous similar messages [ 7062.893226] Lustre: server umount lustre-MDT0000 complete [ 7067.106179] LustreError: 137036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.134413] LustreError: 137036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7067.268905] LustreError: 136139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081627 with bad export cookie 7231782397830882158 [ 7067.291520] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.824863] Lustre: server umount lustre-MDT0001 complete [ 7080.703459] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7083.726481] Key type lgssc unregistered [ 7084.014881] LNet: 147266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7084.023576] LNetError: 147266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7084.040898] LNet: Removed LNI 192.168.201.106@tcp [ 7084.941152] Key type .llcrypt unregistered [ 7084.943226] Key type ._llcrypt unregistered [ 7107.573254] Key type ._llcrypt registered [ 7107.576298] Key type .llcrypt registered [ 7107.675384] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7123.014435] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7124.404705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7124.485198] alg: No test for adler32 (adler32-zlib) [ 7125.615053] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 7126.113799] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 7127.791184] Key type lgssc registered [ 7128.709508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7165.824549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7165.900590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.239779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7167.290223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7167.398969] Lustre: lustre-MDT0000: new disk, initializing [ 7167.460794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7167.471602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7171.674775] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7184.349336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.476703] Lustre: 150469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7184.585889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7184.592606] Lustre: Skipped 1 previous similar message [ 7184.767399] Lustre: lustre-MDT0001: new disk, initializing [ 7185.024681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7185.118040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7185.142768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7191.952221] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7202.860968] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7209.187326] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7216.139875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.371942] Lustre: lustre-OST0000: new disk, initializing [ 7216.374628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7216.378586] Lustre: 152442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7216.446535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.360937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7222.368749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7222.501335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7222.659894] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7232.104861] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7238.956306] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_module obdecho/obdecho [ 7240.826796] LustreError: 153445:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7240.831331] LustreError: 153445:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7242.988343] LustreError: 153542:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7258.595693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7258.611080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7258.619086] Lustre: Skipped 2 previous similar messages [ 7263.412460] Lustre: server umount lustre-MDT0000 complete [ 7268.110494] LustreError: 150463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081828 with bad export cookie 15105047995416134142 [ 7268.114605] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7268.130606] LustreError: 150463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7268.620433] Lustre: server umount lustre-MDT0001 complete [ 7284.145225] Lustre: server umount lustre-OST0000 complete [ 7308.949313] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7321.734115] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7368.851491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.157268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7369.197509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7369.322726] Lustre: lustre-MDT0000: new disk, initializing [ 7369.465357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.492948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7375.170326] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7386.318530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.463237] Lustre: 157259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7386.510470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7386.514549] Lustre: Skipped 1 previous similar message [ 7386.585937] Lustre: lustre-MDT0001: new disk, initializing [ 7386.687502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7386.733583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7386.749321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7391.190433] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7401.484628] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7407.903260] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7415.680317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7415.983843] Lustre: lustre-OST0000: new disk, initializing [ 7415.987720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7415.996436] Lustre: 159232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7416.094609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7422.036298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7422.052966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7422.163961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7423.474528] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7433.621861] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.569122] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7439.837575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7445.471101] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7445.699566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7448.033557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.044745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7448.053588] Lustre: Skipped 3 previous similar messages [ 7448.073377] Lustre: Skipped 1 previous similar message [ 7453.164390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7453.170979] Lustre: Skipped 1 previous similar message [ 7458.286849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7458.299618] Lustre: Skipped 1 previous similar message [ 7462.367731] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7462.649956] Lustre: server umount lustre-OST0000 complete [ 7473.636217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7473.636909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.639805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7473.658619] Lustre: Skipped 1 previous similar message [ 7477.004886] Lustre: server umount lustre-MDT0000 complete [ 7478.752541] LustreError: 157268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7478.778410] LustreError: 157268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7481.407788] LustreError: 159231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082041 with bad export cookie 15105047995416135101 [ 7481.410663] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.416612] LustreError: 159231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7481.815741] Lustre: server umount lustre-MDT0001 complete [ 7485.374941] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:54:03 (1786082043) [ 7487.543785] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7489.956459] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:54:07 (1786082047) [ 7498.849846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.634450] LustreError: 161645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.835050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.872605] LustreError: 161646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.302624] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7508.973216] LustreError: 161645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7514.091265] LustreError: 161646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7514.446919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.369723] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7529.328297] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7534.977897] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7543.639255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.231269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7544.233338] Lustre: Skipped 1 previous similar message [ 7552.359807] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7562.408218] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7569.892320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.902316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7569.910181] Lustre: Skipped 1 previous similar message [ 7569.926344] Lustre: Skipped 3 previous similar messages [ 7575.658804] Lustre: server umount lustre-MDT0000 complete [ 7578.732189] LustreError: 163087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.805788] LustreError: 161627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082139 with bad export cookie 15105047995416135983 [ 7579.812057] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7579.820188] LustreError: 161627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7580.138240] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.149227] Lustre: Skipped 1 previous similar message [ 7586.400895] Lustre: server umount lustre-MDT0001 complete [ 7593.406098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7608.841555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.076664] LustreError: 165489:0:(llog.c:1655:llog_backup()) MGC192.168.201.106@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7609.084321] Lustre: 165489:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7609.832689] LustreError: 165494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7609.847835] LustreError: 165494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7644.641768] LustreError: 165496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.671039] LustreError: 165496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7649.795989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7649.800973] Lustre: Skipped 7 previous similar messages [ 7670.239322] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7670.243939] Lustre: 165489:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7709.668552] LustreError: 163479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.709030] LustreError: 163479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7711.238956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7715.882100] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7716.327625] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7725.338235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.882125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7754.738895] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7755.154830] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7763.354156] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7765.817896] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7772.913030] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7775.459408] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7796.974307] Lustre: server umount lustre-MDT0000 complete [ 7800.802749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.808862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7802.443367] LustreError: 161626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082362 with bad export cookie 15105047995416142444 [ 7802.446181] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7802.466095] LustreError: 161626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7803.125840] Lustre: server umount lustre-MDT0001 complete [ 7818.914966] Lustre: server umount lustre-OST0000 complete [ 7844.933510] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7857.740287] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7898.260493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.524389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7898.542558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7898.610262] Lustre: lustre-MDT0000: new disk, initializing [ 7898.676546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.679585] Lustre: Skipped 1 previous similar message [ 7898.698360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7903.644363] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7915.111526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7915.260059] Lustre: 170599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7915.547510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7920.430494] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7930.305839] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7936.426780] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7944.304102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7944.507495] Lustre: lustre-OST0000: new disk, initializing [ 7944.509906] Lustre: Skipped 1 previous similar message [ 7944.512590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7944.515989] Lustre: Skipped 2 previous similar messages [ 7944.519881] Lustre: 172578:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7952.258107] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7954.138422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7954.149995] Lustre: Skipped 1 previous similar message [ 7954.154508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7954.296929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7963.694600] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7969.602854] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7969.836830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7975.911922] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7976.291475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7980.000297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7980.009218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.029789] Lustre: Skipped 2 previous similar messages [ 7980.033275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7993.311342] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7993.517179] Lustre: server umount lustre-OST0000 complete [ 8012.255266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8012.680505] Lustre: server umount lustre-MDT0000 complete [ 8015.841550] LustreError: 170612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8015.876780] LustreError: 170612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 8017.136836] LustreError: 170593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082577 with bad export cookie 15105047995416159272 [ 8017.145481] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8017.158143] LustreError: 170593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8017.616641] Lustre: server umount lustre-MDT0001 complete [ 8027.246056] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:03:05 (1786082585) [ 8035.046046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.249410] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8051.981344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8057.809518] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8065.225320] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8071.136583] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8077.908710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8084.972627] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8092.515435] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8105.291870] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:04:23 (1786082663) [ 8111.074969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8111.089912] LustreError: Skipped 1 previous similar message [ 8111.100315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.113151] Lustre: Skipped 2 previous similar messages [ 8111.125744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8111.131574] Lustre: Skipped 11 previous similar messages [ 8116.469897] Lustre: server umount lustre-OST0000 complete [ 8127.067939] Lustre: server umount lustre-MDT0000 complete [ 8131.463522] LustreError: 174974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082691 with bad export cookie 15105047995416160154 [ 8131.480812] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8132.030658] Lustre: server umount lustre-MDT0001 complete [ 8145.253216] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8149.443866] Key type lgssc unregistered [ 8149.726299] LNet: 178727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8149.743968] LNetError: 178727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8149.771991] LNet: Removed LNI 192.168.201.106@tcp [ 8150.789704] Key type .llcrypt unregistered [ 8150.797819] Key type ._llcrypt unregistered [ 8173.651555] Key type ._llcrypt registered [ 8173.656898] Key type .llcrypt registered [ 8173.788716] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8174.755212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8175.126985] alg: No test for adler32 (adler32-zlib) [ 8176.345314] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8176.602103] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8178.327240] Key type lgssc registered [ 8179.927546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8188.816862] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8188.835775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8204.511805] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8209.639213] LustreError: 179971:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [ 8214.816366] LustreError: 179971:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8214.831684] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8214.843800] LustreError: MGC192.168.201.106@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8214.857843] LustreError: 179971:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8214.869207] LustreError: 179971:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8214.888309] LustreError: 179971:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8214.951821] Lustre: server umount lustre-OST0000 complete [ 8214.953821] LustreError: 179971:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8225.261615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.707588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8231.636793] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8236.007743] LustreError: 180471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.030811] LustreError: 180471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8240.037294] LustreError: 180470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.141696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.609025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8248.444420] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8258.935574] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8267.084442] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8274.433588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8274.546081] LustreError: 182286:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8274.550460] Lustre: 182286:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.106@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8274.623968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8281.438541] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8292.146886] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8297.443305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.459037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.459661] Lustre: Skipped 2 previous similar messages [ 8297.490036] Lustre: Skipped 2 previous similar messages [ 8302.567260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8302.584904] Lustre: Skipped 2 previous similar messages [ 8303.213483] Lustre: server umount lustre-MDT0000 complete [ 8307.682308] LustreError: 180465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.706877] LustreError: 180465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8308.034473] LustreError: 180451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082868 with bad export cookie 8956803098067326809 [ 8308.063132] LustreError: 180451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8308.075070] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8308.818985] Lustre: server umount lustre-MDT0001 complete [ 8323.776595] Lustre: server umount lustre-OST0000 complete [ 8333.632912] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8347.443137] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8392.947751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.252056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8393.319525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8393.465801] Lustre: lustre-MDT0000: new disk, initializing [ 8393.575311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.596052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8399.830393] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8412.625209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.706554] Lustre: 185819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8412.755879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8412.759786] Lustre: Skipped 1 previous similar message [ 8412.834317] Lustre: lustre-MDT0001: new disk, initializing [ 8412.934826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8413.017977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8413.040244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8419.579408] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8432.132117] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8437.709541] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8445.436031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8445.844183] Lustre: lustre-OST0000: new disk, initializing [ 8445.851951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8445.876161] Lustre: 187789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8446.086304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.001947] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8455.760835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8455.767531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8455.908641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8464.311665] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8469.047540] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8469.331728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8474.424663] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8474.663699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8481.765142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8481.773506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.781818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8481.783993] Lustre: Skipped 1 previous similar message [ 8482.937987] Lustre: server umount lustre-OST0000 complete [ 8489.959070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8489.968499] LustreError: Skipped 1 previous similar message [ 8489.970468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.993173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8490.002028] Lustre: Skipped 1 previous similar message [ 8492.010765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.499929] Lustre: server umount lustre-MDT0000 complete [ 8497.130033] LustreError: 185827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8497.164640] LustreError: 185827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8497.989215] LustreError: 185812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083058 with bad export cookie 8956803098067327523 [ 8497.993216] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8498.004350] LustreError: 185812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8498.022753] LustreError: 179311:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 8498.435067] Lustre: server umount lustre-MDT0001 complete [ 8508.410128] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:11:06 (1786083066) [ 8526.791335] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8530.261271] Key type lgssc unregistered [ 8530.725977] LNet: 190736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8530.732807] LNetError: 190736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8530.753353] LNet: Removed LNI 192.168.201.106@tcp [ 8532.092347] Key type .llcrypt unregistered [ 8532.096657] Key type ._llcrypt unregistered [ 8553.221333] Key type ._llcrypt registered [ 8553.224275] Key type .llcrypt registered [ 8553.354626] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8554.658166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8554.785846] alg: No test for adler32 (adler32-zlib) [ 8555.833550] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8556.000852] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8557.656305] Key type lgssc registered [ 8559.001374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8567.383804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8567.417079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.952514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.029719] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8579.566877] LustreError: 191954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8579.590795] LustreError: 191954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8584.156591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.505601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8589.395824] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8596.731776] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8602.880974] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8612.482493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8612.871819] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8612.874371] LustreError: 193818:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8612.880413] Lustre: 193818:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.106@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8613.062397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8623.345725] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8633.173485] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8635.884438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8635.900075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8635.913752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.970321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.971739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.980392] Lustre: Skipped 1 previous similar message [ 8640.003935] Lustre: Skipped 1 previous similar message [ 8641.817450] Lustre: server umount lustre-MDT0000 complete [ 8645.095422] LustreError: 191950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8645.136539] LustreError: 191950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8646.417636] LustreError: 191936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083206 with bad export cookie 9697524161790053434 [ 8646.429985] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8646.923955] Lustre: server umount lustre-MDT0001 complete [ 8663.009288] Lustre: server umount lustre-OST0000 complete [ 8676.096725] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8692.433252] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8735.727623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.113638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8736.151996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8736.298239] Lustre: lustre-MDT0000: new disk, initializing [ 8736.453097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.484187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8741.667123] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8754.194721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.345600] Lustre: 197369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8754.393869] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8754.397776] Lustre: Skipped 1 previous similar message [ 8754.400900] LustreError: 198151:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8754.407242] LustreError: 198151:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8754.412141] Lustre: 198151:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8754.428610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8754.437093] Lustre: Skipped 1 previous similar message [ 8754.642378] Lustre: lustre-MDT0001: new disk, initializing [ 8754.862385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8754.928596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8754.940521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8760.581165] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8773.050609] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8780.385377] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8788.363763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8788.676216] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8788.680732] Lustre: Skipped 1 previous similar message [ 8788.684705] LustreError: 199315:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8788.694152] LustreError: 199315:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8788.699492] Lustre: 199315:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8788.706816] Lustre: 199315:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8788.728848] Lustre: lustre-OST0000: new disk, initializing [ 8788.732711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8788.739050] Lustre: 199338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8788.902148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8796.350464] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8798.349143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8798.363438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8798.510264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8807.795558] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8813.613950] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8813.824309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8819.527125] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8819.821642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8824.288685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8824.296218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.303563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.309107] Lustre: Skipped 2 previous similar messages [ 8828.507320] Lustre: server umount lustre-OST0000 complete [ 8834.529272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.557158] Lustre: Skipped 1 previous similar message [ 8834.572774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8834.588230] Lustre: Skipped 1 previous similar message [ 8839.655523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8839.669319] Lustre: Skipped 1 previous similar message [ 8847.327176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8847.484238] Lustre: server umount lustre-MDT0000 complete [ 8849.899269] LustreError: 198873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8849.923775] LustreError: 198873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8852.138914] LustreError: 197362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083412 with bad export cookie 9697524161790054162 [ 8852.167364] LustreError: 197362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8852.174473] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.848988] Lustre: server umount lustre-MDT0001 complete [ 8864.109859] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:17:02 (1786083422) [ 8886.256860] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8899.268917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.813816] LustreError: 202469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8899.862937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.939869] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8905.190572] LustreError: 202470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8914.402967] LustreError: 202469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8914.410121] LustreError: 202469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8915.501714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.538634] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8923.294803] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8930.450484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8937.743322] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8947.847852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8948.048677] Lustre: lustre-OST0001: new disk, initializing [ 8948.051848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8948.069175] Lustre: 204774:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8948.200944] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8948.204445] Lustre: Skipped 2 previous similar messages [ 8949.650115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8949.662813] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8949.832028] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8954.485737] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8971.504787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8981.105683] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8985.213101] Lustre: DEBUG MARKER: ADD OST3 [ 8991.883202] loop: module loaded [ 8992.020825] loop0: detected capacity change from 0 to 204800000 [ 8992.329726] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8992.342371] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8992.358417] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8999.928685] Lustre: DEBUG MARKER: STOP OST3 [ 9000.724095] loop0: detected capacity change from 0 to 204800000 [ 9008.582394] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.791345] Lustre: lustre-OST0003: new disk, initializing [ 9008.794782] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9008.799996] Lustre: 206865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9010.562716] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9010.792679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9010.797752] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9010.833978] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9011.150063] Lustre: Failing over lustre-OST0003 [ 9011.361487] Lustre: server umount lustre-OST0003 complete [ 9012.580139] LustreError: 207000:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9015.453333] LustreError: 204784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.788220] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9015.802124] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9015.806092] Lustre: Skipped 1 previous similar message [ 9015.824533] LustreError: Skipped 1 previous similar message [ 9017.359779] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9029.124267] Lustre: DEBUG MARKER: START OST3 again [ 9036.050523] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9036.300385] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9036.309332] Lustre: Skipped 1 previous similar message [ 9036.322856] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9038.018906] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9038.135283] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9038.137149] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9043.355697] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9064.294264] Lustre: Failing over lustre-OST0003 [ 9064.396855] Lustre: server umount lustre-OST0003 complete [ 9066.994332] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9067.016404] Lustre: Skipped 1 previous similar message [ 9067.021516] LustreError: 203955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.039600] LustreError: 203955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 9084.899238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9084.907978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9084.933732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9084.937338] Lustre: Skipped 2 previous similar messages [ 9089.421222] Lustre: server umount lustre-MDT0000 complete [ 9093.163762] LustreError: 202449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083653 with bad export cookie 9697524161790055044 [ 9093.174266] LustreError: 202449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9093.176891] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9099.657691] Lustre: server umount lustre-MDT0001 complete [ 9104.143312] Lustre: server umount lustre-OST0000 complete [ 9108.786791] Lustre: server umount lustre-OST0001 complete [ 9116.949124] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9126.735619] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9164.811046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.028308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9165.058658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9165.121649] Lustre: lustre-MDT0000: new disk, initializing [ 9165.190138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9165.205500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9169.837448] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9182.810491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.069572] Lustre: 212394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9183.187564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9183.192150] Lustre: Skipped 1 previous similar message [ 9183.464881] Lustre: lustre-MDT0001: new disk, initializing [ 9183.655121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9183.666086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9190.739334] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9204.519410] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9211.445855] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9219.279649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9219.551512] Lustre: lustre-OST0000: new disk, initializing [ 9219.556215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9219.564703] Lustre: 214364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9221.489704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9221.499992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9221.608851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9225.701106] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9235.675425] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9241.210904] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9241.461056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9246.158707] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9246.400884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9252.336380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9252.348642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9252.390868] Lustre: Skipped 5 previous similar messages [ 9252.397702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9252.414303] Lustre: Skipped 9 previous similar messages [ 9255.343835] Lustre: server umount lustre-OST0000 complete [ 9267.071478] Lustre: server umount lustre-MDT0000 complete [ 9267.681668] LustreError: 212407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9267.692727] LustreError: 212407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 9271.464482] LustreError: 212388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083831 with bad export cookie 9697524161790057340 [ 9271.480752] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9271.864240] Lustre: server umount lustre-MDT0001 complete [ 9276.199843] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:23:54 (1786083834) [ 9293.910402] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9304.874827] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9350.706741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9350.920917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9350.946384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9351.011757] Lustre: lustre-MDT0000: new disk, initializing [ 9351.079749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9355.673649] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9365.435646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.635701] Lustre: 218976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9365.992781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9371.231925] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9382.310700] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9388.091567] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9397.048248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9397.710370] Lustre: 220948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9399.590918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9399.695614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9406.554286] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9417.333896] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9422.737543] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9422.926143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9427.912136] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9428.111745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9430.500962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9430.513116] Lustre: Skipped 3 previous similar messages [ 9430.519977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9430.522761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9430.525840] Lustre: Skipped 2 previous similar messages [ 9430.535075] LustreError: Skipped 1 previous similar message [ 9444.319183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9444.566190] Lustre: server umount lustre-OST0000 complete [ 9455.715755] Lustre: server umount lustre-MDT0000 complete [ 9456.100876] LustreError: 218989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9456.118429] LustreError: 218989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9459.406463] LustreError: 220947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084019 with bad export cookie 9697524161790058222 [ 9459.411356] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9459.420578] LustreError: 220947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9459.816319] Lustre: server umount lustre-MDT0001 complete [ 9468.318261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.998734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9469.007918] Lustre: Skipped 5 previous similar messages [ 9474.180822] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9482.181990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.651351] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9495.471499] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9502.694383] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9510.246434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9519.054790] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9528.675395] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9540.577352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9540.595282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9540.601034] Lustre: Skipped 7 previous similar messages [ 9545.429629] Lustre: server umount lustre-OST0000 complete [ 9555.486728] Lustre: server umount lustre-MDT0000 complete [ 9559.503370] LustreError: 223146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084119 with bad export cookie 9697524161790059104 [ 9559.507868] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9559.512760] LustreError: 223146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9559.991223] Lustre: server umount lustre-MDT0001 complete [ 9669.251957] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9672.099556] Key type lgssc unregistered [ 9672.357040] LNet: 226830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9672.362543] LNetError: 226830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9672.373830] LNet: Removed LNI 192.168.201.106@tcp [ 9673.122314] Key type .llcrypt unregistered [ 9673.124312] Key type ._llcrypt unregistered [ 9702.164193] Key type ._llcrypt registered [ 9702.168702] Key type .llcrypt registered [ 9702.323705] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9719.709498] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9721.400797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9721.460774] alg: No test for adler32 (adler32-zlib) [ 9722.594871] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 9722.785067] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9724.447196] Key type lgssc registered [ 9725.617927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9771.106138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9771.158321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.535446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9772.612248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9772.811355] Lustre: lustre-MDT0000: new disk, initializing [ 9772.988498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9773.038629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9778.350948] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9789.727845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9789.825471] Lustre: 230188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9789.893951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9789.899185] Lustre: Skipped 1 previous similar message [ 9789.991514] Lustre: lustre-MDT0001: new disk, initializing [ 9790.106493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9790.154797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9790.179527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9795.756692] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9807.904795] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9815.146321] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9822.741522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9823.046761] Lustre: lustre-OST0000: new disk, initializing [ 9823.067303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9823.078731] Lustre: 232155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9823.164263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9830.255671] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9831.532251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9831.552584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9831.678541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9842.968533] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9848.970488] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9849.327233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9855.180955] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9855.454956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9862.627890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9862.628782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9862.656479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9862.662330] LustreError: Skipped 1 previous similar message [ 9862.675499] Lustre: Skipped 1 previous similar message [ 9864.561273] Lustre: server umount lustre-OST0000 complete [ 9872.355883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9872.369314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9872.376398] Lustre: Skipped 1 previous similar message [ 9872.382967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9875.860275] Lustre: server umount lustre-MDT0000 complete [ 9880.272411] LustreError: 230181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084440 with bad export cookie 5295134477693682499 [ 9880.287648] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9880.289342] LustreError: 230181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9880.707401] Lustre: server umount lustre-MDT0001 complete [ 9901.350709] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9913.168627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.621640] LustreError: 235091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9913.634133] LustreError: 235091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9913.691656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.949200] LustreError: 235092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9919.008650] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9923.043643] LustreError: 235091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.165240] LustreError: 235092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.399784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9928.932109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9935.300247] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9939.458876] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9947.445509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9947.902814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.162371] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9964.421568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9964.545846] Lustre: lustre-OST0001: new disk, initializing [ 9964.548186] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9964.551240] Lustre: 237399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9964.639557] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9971.323653] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9974.417470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9974.434088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9974.560397] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9982.900435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9992.981677] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10008.577769] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:36:06 (1786084566) [10015.715380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.722169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10015.733408] Lustre: Skipped 4 previous similar messages [10015.754368] Lustre: Skipped 4 previous similar messages [10017.833257] Lustre: server umount lustre-MDT0000 complete [10020.836490] LustreError: 235091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10020.863725] LustreError: 235091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10022.174027] LustreError: 237398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084582 with bad export cookie 5295134477693683381 [10022.190905] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10022.192750] LustreError: 237398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10022.626649] Lustre: server umount lustre-MDT0001 complete [10037.860697] Lustre: server umount lustre-OST0000 complete [10051.264795] Lustre: server umount lustre-OST0001 complete [10060.998544] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [10073.922574] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10116.111736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.364990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10116.397847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10116.480902] Lustre: lustre-MDT0000: new disk, initializing [10116.556868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10116.577573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10122.918226] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10136.768864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10136.948981] Lustre: 241349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10137.060874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10137.067358] Lustre: Skipped 1 previous similar message [10137.167972] Lustre: lustre-MDT0001: new disk, initializing [10137.406341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10137.418861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10143.985448] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10157.595747] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10164.546734] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10173.722797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10174.226210] Lustre: lustre-OST0000: new disk, initializing [10174.229842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10174.237746] Lustre: 243319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10175.779932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10175.790801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10175.971082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10183.047165] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10194.685884] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10199.731206] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10199.981699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10206.504603] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10206.865525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10211.825253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10211.834570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10211.846147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10211.848338] Lustre: Skipped 1 previous similar message [10215.751584] Lustre: server umount lustre-OST0000 complete [10222.052173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10222.069488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10222.072277] Lustre: Skipped 1 previous similar message [10227.838674] Lustre: server umount lustre-MDT0000 complete [10231.608818] LustreError: 241343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084791 with bad export cookie 5295134477693685516 [10231.610167] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10231.640028] LustreError: 241343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10232.177535] Lustre: server umount lustre-MDT0001 complete [10241.932242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10242.470115] LustreError: 245536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.636914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10242.644696] Lustre: Skipped 2 previous similar messages [10248.314322] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10257.281262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.471881] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10271.373901] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10279.015937] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10286.928697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10294.403838] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10302.998875] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10451.432304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10451.445364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10451.459144] Lustre: Skipped 1 previous similar message [10451.468049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10451.472839] Lustre: Skipped 3 previous similar messages [10456.813902] Lustre: server umount lustre-OST0000 complete [10467.178761] Lustre: server umount lustre-MDT0000 complete [10471.150057] LustreError: 245518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085031 with bad export cookie 5295134477693686398 [10471.172990] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10471.709476] Lustre: server umount lustre-MDT0001 complete [10573.650507] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10576.970769] Key type lgssc unregistered [10577.268259] LNet: 249156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10577.279700] LNetError: 249156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10577.307677] LNet: Removed LNI 192.168.201.106@tcp [10579.026175] Key type .llcrypt unregistered [10579.034449] Key type ._llcrypt unregistered [10611.366011] Key type ._llcrypt registered [10611.369169] Key type .llcrypt registered [10611.527860] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [10629.463982] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10631.073096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10631.102767] alg: No test for adler32 (adler32-zlib) [10632.161552] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [10632.418878] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10634.063190] Key type lgssc registered [10635.497847] Lustre: Echo OBD driver; http://www.lustre.org/ [10689.718316] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10703.198678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10703.266432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.657376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10704.693801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10704.804377] Lustre: lustre-MDT0000: new disk, initializing [10704.882867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10704.897388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10711.164674] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10724.740575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.005279] Lustre: 253233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10725.177334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10725.180158] Lustre: Skipped 1 previous similar message [10725.300607] Lustre: lustre-MDT0001: new disk, initializing [10725.452165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10725.520263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10725.537079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10731.652606] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10737.739390] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10745.799836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10746.239633] Lustre: lustre-OST0000: new disk, initializing [10746.244090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10746.256098] Lustre: 254866:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10746.370192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10749.609624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10749.619697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10749.765288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10754.331914] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10766.052494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10766.209402] Lustre: lustre-OST0001: new disk, initializing [10766.213265] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10766.222389] Lustre: 255737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10766.337137] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10773.031641] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10775.609321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10775.620856] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10775.717552] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10786.397779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10795.976440] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10811.887734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.892910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10811.909481] Lustre: Skipped 1 previous similar message [10811.930172] Lustre: Skipped 3 previous similar messages [10816.786201] Lustre: server umount lustre-MDT0000 complete [10820.535680] LustreError: 254865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085380 with bad export cookie 4413588690308982797 [10820.539371] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.550899] LustreError: 254865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10821.042845] Lustre: server umount lustre-MDT0001 complete [10835.715353] Lustre: server umount lustre-OST0000 complete [10837.984030] Lustre: 250740:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786085382/real 1786085382] req@ffff9edfc8ecb100 x1872846066367488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786085398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10838.029376] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10838.055960] Lustre: Skipped 2 previous similar messages [10840.142348] Lustre: server umount lustre-OST0001 complete [10859.880852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10871.203145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10880.478237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10893.197503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10914.457851] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10925.785955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.802484] Lustre: lustre-MDT0000: reset Object Index mappings [10926.218910] LustreError: 260748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10926.237592] LustreError: 260748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10926.330100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10930.892290] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10931.682139] LustreError: 260749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10936.810278] LustreError: 260748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.694960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10939.708632] Lustre: lustre-MDT0001: reset Object Index mappings [10939.983995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10944.166660] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10947.183770] Lustre: 261890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10953.547521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10962.537551] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10965.547612] LustreError: 262245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.608951] LustreError: 262246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.634145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10973.332094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10973.762781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10973.767597] Lustre: Skipped 1 previous similar message [10978.859148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10980.907318] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10989.132795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10993.118485] Lustre: 263764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10999.266872] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.289891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10999.303904] Lustre: Skipped 3 previous similar messages [11000.483655] Lustre: server umount lustre-MDT0000 complete [11004.390584] LustreError: 260745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11004.417256] LustreError: 260745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11004.934236] LustreError: 263076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085565 with bad export cookie 4413588690308989587 [11004.945416] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11004.967338] LustreError: 263076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11005.298159] Lustre: server umount lustre-MDT0001 complete [11020.418076] Lustre: server umount lustre-OST0000 complete [11034.761943] Lustre: server umount lustre-OST0001 complete [11057.465410] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:53:35 (1786085615) [11075.745716] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [11086.515963] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11128.767522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11128.944308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11128.976206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11129.039990] Lustre: lustre-MDT0000: new disk, initializing [11129.082381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11129.089942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11135.215659] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11148.050773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11148.215513] Lustre: 268022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11148.252789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11148.254805] Lustre: Skipped 1 previous similar message [11148.338263] Lustre: lustre-MDT0001: new disk, initializing [11148.479685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11148.483838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11152.614870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11164.738842] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11171.242899] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11179.250289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11179.677873] Lustre: lustre-OST0000: new disk, initializing [11179.682617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11179.689539] Lustre: 269990:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11181.718110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11181.733195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11181.924487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11187.519192] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11199.752725] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11205.579782] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11205.883896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11211.060421] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11211.330765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11217.899485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11217.907593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11217.916393] LustreError: Skipped 1 previous similar message [11217.917419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11217.954643] Lustre: Skipped 4 previous similar messages [11219.604442] Lustre: server umount lustre-OST0000 complete [11225.568462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11225.575376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11225.584870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11225.588826] Lustre: Skipped 1 previous similar message [11230.521906] Lustre: server umount lustre-MDT0000 complete [11233.248491] LustreError: 268028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11233.259217] LustreError: 268028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11234.570569] LustreError: 268014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085794 with bad export cookie 4413588690308991673 [11234.574346] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11234.577407] LustreError: 268014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11234.855546] Lustre: server umount lustre-MDT0001 complete [11261.494120] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11276.885786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.748671] LustreError: 272949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11277.870547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11277.880409] Lustre: Skipped 2 previous similar messages [11282.053831] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11291.544929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11296.948327] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11300.075551] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11307.939754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11317.637417] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11327.874042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11328.018552] Lustre: lustre-OST0001: new disk, initializing [11328.021646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11328.031799] Lustre: 275259:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11329.408103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11329.411703] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11329.452948] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11334.500820] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11349.249553] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11354.380255] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11361.724542] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11361.750542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11374.065857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11374.081397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11374.102816] Lustre: Skipped 1 previous similar message [11374.107636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11374.116505] Lustre: Skipped 2 previous similar messages [11378.091609] Lustre: server umount lustre-MDT0000 complete [11380.707427] LustreError: 272949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.715256] LustreError: 272949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11382.845886] LustreError: 275258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085942 with bad export cookie 4413588690308992562 [11382.855113] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11382.857543] LustreError: 275258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11383.410699] Lustre: server umount lustre-MDT0001 complete [11399.091602] Lustre: server umount lustre-OST0000 complete [11400.991145] Lustre: 250740:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786085945/real 1786085945] req@ffff9edfcb78c380 x1872846066540544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786085961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11401.008587] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11401.021628] Lustre: Skipped 3 previous similar messages [11403.231119] Lustre: 250738:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786085947/real 1786085947] req@ffff9edfcb78d500 x1872846066540800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786085963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11403.685717] Lustre: server umount lustre-OST0001 complete [11423.823652] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11436.467251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.966415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11436.977989] Lustre: Skipped 3 previous similar messages [11442.770586] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11452.971698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11458.427089] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11463.350942] Lustre: 279459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11472.218238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11479.749820] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11482.881703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11489.629051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11495.440274] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11497.407450] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11511.206639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11516.108817] Lustre: 281349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11524.923582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11525.227368] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11537.743127] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:01:35 (1786086095) [11555.810034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11555.820595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.840950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.851295] Lustre: Skipped 4 previous similar messages [11561.847562] Lustre: server umount lustre-MDT0000 complete [11561.977203] LustreError: 279315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11561.999197] LustreError: 279315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11566.572783] LustreError: 278301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086126 with bad export cookie 4413588690308999345 [11566.590899] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.591828] LustreError: 278301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11573.310699] Lustre: server umount lustre-MDT0001 complete [11593.183199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11593.260873] Lustre: server umount lustre-OST0000 complete [11597.723732] Lustre: server umount lustre-OST0001 complete [11604.952881] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [11616.758732] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11675.541226] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11690.718420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.133307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11691.231686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11691.350291] Lustre: lustre-MDT0000: new disk, initializing [11691.662372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11697.391746] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11710.733484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.884484] Lustre: 285596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11711.181927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11711.190284] Lustre: Skipped 4 previous similar messages [11711.221921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11716.045527] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11731.616956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11732.263558] Lustre: 287231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11734.453980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11742.060591] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11753.647748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11753.914716] Lustre: 288101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11755.835736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11755.845239] Lustre: Skipped 1 previous similar message [11756.035479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11762.799090] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11774.008814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11777.786247] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11777.790228] Lustre: Skipped 1 previous similar message [11800.563843] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:05:57 (1786086357) [11803.675758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11803.681962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.705113] Lustre: Skipped 5 previous similar messages [11803.712120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11803.716248] Lustre: Skipped 8 previous similar messages [11810.385344] Lustre: server umount lustre-MDT0000 complete [11814.226087] LustreError: 285587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086374 with bad export cookie 4413588690309006807 [11814.259830] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11814.786490] Lustre: server umount lustre-MDT0001 complete [11830.592986] Lustre: server umount lustre-OST0000 complete [11833.826952] Lustre: 250740:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786086377/real 1786086377] req@ffff9ee100931500 x1872846066709632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786086393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11835.068490] Lustre: server umount lustre-OST0001 complete [11844.856126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.330145] LustreError: 290350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.355708] LustreError: 290350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11850.637833] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11860.563758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.651219] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11876.783168] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11885.421841] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11894.166675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11901.444396] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11909.316289] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.947801] loop1: detected capacity change from 0 to 204800000 [11934.233381] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11934.240190] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11934.252309] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11941.056459] loop1: detected capacity change from 0 to 204800000 [11948.363159] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.477954] LustreError: 290340:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs11840: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11948.485431] LustreError: 290340:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs11840 [11948.489912] LustreError: 290340:0:(mgs_handler.c:517:mgs_target_reg()) fs11840-MDT0001: Error checking target index: rc = -13 [11948.500046] LustreError: fs11840-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11948.507599] LustreError: 294070:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11948.512966] LustreError: 294070:0:(tgt_mount.c:2129:server_put_super()) no obd fs11840-MDT0001 [11948.516730] LustreError: 294070:0:(tgt_mount.c:116:server_deregister_mount()) fs11840-MDT0001 not registered [11948.542344] Lustre: server umount fs11840-MDT0001 complete [11948.545477] LustreError: 294070:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11958.250522] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.397163] Lustre: Setting parameter fs11840-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs11840-MDT0001 [11958.479779] Lustre: srv-fs11840-MDT0001: No data found on store. Initialize space. [11958.485345] Lustre: Skipped 4 previous similar messages [11958.544844] Lustre: fs11840-MDT0001: new disk, initializing [11958.547100] Lustre: Skipped 3 previous similar messages [11958.650897] Lustre: srv-fs11840-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11958.661964] Lustre: 294536:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11840-MDT0001-mdtlov: can't create new llog: rc = -115 [11958.670071] LustreError: 294536:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11840-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11965.176316] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11972.626195] Lustre: server umount fs11840-MDT0001 complete [11986.932959] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11988.549139] loop1: detected capacity change from 0 to 204800000 [11996.039226] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11996.930487] Lustre: 296107:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11840-MDT0001-mdtlov: can't create new llog: rc = -115 [11996.943184] LustreError: 296107:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11840-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12002.527674] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12009.363339] Lustre: server umount fs11840-MDT0001 complete [12022.759658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.761408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.781508] Lustre: Skipped 6 previous similar messages [12022.805183] Lustre: Skipped 6 previous similar messages [12026.961641] Lustre: server umount lustre-MDT0000 complete [12035.991255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12036.156279] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12041.103780] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12041.707947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12048.869913] loop1: detected capacity change from 0 to 204800000 [12057.625888] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12058.388071] Lustre: 298552:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11840-MDT0001-mdtlov: can't create new llog: rc = -115 [12058.405834] LustreError: 298552:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11840-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12065.490209] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12073.746572] Lustre: server umount fs11840-MDT0001 complete [12103.138867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12103.160515] LustreError: Skipped 1 previous similar message [12103.216036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12103.227516] Lustre: Skipped 2 previous similar messages [12107.388655] Lustre: server umount lustre-OST0000 complete [12118.529130] Lustre: server umount lustre-MDT0000 complete [12123.081985] LustreError: 290332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086683 with bad export cookie 4413588690309009488 [12123.088939] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12123.096602] LustreError: 290332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12123.547099] Lustre: server umount lustre-MDT0001 complete [12138.560885] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12141.249580] Key type lgssc unregistered [12141.605211] LNet: 300901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12141.613466] LNetError: 300901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12142.636110] LNet: Removed LNI 192.168.201.106@tcp [12143.549157] Key type .llcrypt unregistered [12143.551057] Key type ._llcrypt unregistered [12165.672712] Key type ._llcrypt registered [12165.674751] Key type .llcrypt registered [12165.796904] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12167.102183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12167.281655] alg: No test for adler32 (adler32-zlib) [12168.516564] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [12168.875812] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12170.615294] Key type lgssc registered [12172.287267] Lustre: Echo OBD driver; http://www.lustre.org/ [12180.808878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12180.829643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.650586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12189.071826] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12192.240246] LustreError: 302190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.248401] LustreError: 302190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12197.365210] LustreError: 302191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12199.496451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12199.872557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12207.856210] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12221.934417] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12228.954887] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12235.715708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12236.084812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12243.234747] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12244.293533] LustreError: 304022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.595635] LustreError: 304023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.911528] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12255.657179] LustreError: 304022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.685132] LustreError: 304022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12266.110299] LustreError: 304022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12266.133789] LustreError: 304022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12274.644940] loop1: detected capacity change from 0 to 204800000 [12275.023349] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12275.045571] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12275.059227] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12281.778950] loop1: detected capacity change from 0 to 204800000 [12283.374838] LustreError: 304023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.419725] LustreError: 304023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [12292.055954] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12292.322289] Lustre: fs14389-OST0000: new disk, initializing [12292.340800] Lustre: srv-fs14389-OST0000: No data found on store. Initialize space. [12292.347817] Lustre: 305932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12292.473472] Lustre: fs14389-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12301.218910] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12311.913448] Lustre: server umount fs14389-OST0000 complete [12317.303763] LustreError: 304022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12317.331268] LustreError: 304022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [12324.333988] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12325.262788] loop1: detected capacity change from 0 to 204800000 [12329.448836] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12331.485324] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12331.691192] Lustre: fs14389-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12339.177859] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12349.052737] loop2: detected capacity change from 0 to 204800000 [12349.369205] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12349.389765] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12349.412835] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12355.745805] loop2: detected capacity change from 0 to 204800000 [12361.957265] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12362.062874] LustreError: fs14389-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12362.070266] LustreError: fs14389-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12362.076727] LustreError: 308974:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12362.081465] LustreError: 308974:0:(tgt_mount.c:2129:server_put_super()) no obd fs14389-OST0001 [12362.085884] LustreError: 308974:0:(tgt_mount.c:116:server_deregister_mount()) fs14389-OST0001 not registered [12362.120403] Lustre: server umount fs14389-OST0001 complete [12362.122137] LustreError: 308974:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [12373.481136] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12373.607278] Lustre: fs14389-OST0001: new disk, initializing [12373.611932] Lustre: srv-fs14389-OST0001: No data found on store. Initialize space. [12373.622318] Lustre: 309524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12373.713653] Lustre: fs14389-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12379.592084] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12383.853415] LustreError: 304023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12383.891505] LustreError: 304023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [12390.788324] Lustre: server umount fs14389-OST0000 complete [12405.158915] Lustre: server umount fs14389-OST0001 complete [12422.442508] loop1: detected capacity change from 0 to 204800000 [12429.036887] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12429.426790] Lustre: fs14389-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12436.384598] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12444.326372] loop2: detected capacity change from 0 to 204800000 [12451.221830] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12458.779595] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12470.625783] Lustre: server umount fs14389-OST0000 complete [12477.929711] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12481.245602] Lustre: server umount fs14389-OST0001 complete [12483.077547] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12498.407700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12498.450407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12498.468263] Lustre: Skipped 1 previous similar message [12503.376453] Lustre: server umount lustre-OST0000 complete [12508.648759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12508.670368] Lustre: Skipped 1 previous similar message [12508.686382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12508.699753] Lustre: Skipped 1 previous similar message [12513.762496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12513.770076] Lustre: Skipped 1 previous similar message [12514.245207] Lustre: server umount lustre-MDT0000 complete [12517.868615] LustreError: 302171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087077 with bad export cookie 4676131651497424924 [12517.870035] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12517.884597] LustreError: 302171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12518.268531] Lustre: server umount lustre-MDT0001 complete [12531.391217] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12534.236946] Key type lgssc unregistered [12534.630790] LNet: 315451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12534.656641] LNetError: 315451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12534.688909] LNet: Removed LNI 192.168.201.106@tcp [12535.557195] Key type .llcrypt unregistered [12535.566905] Key type ._llcrypt unregistered [12571.077484] Key type ._llcrypt registered [12571.078497] Key type .llcrypt registered [12571.145431] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [12592.506546] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12594.109843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12594.394955] alg: No test for adler32 (adler32-zlib) [12595.846727] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [12596.039071] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12597.751422] Key type lgssc registered [12598.788782] Lustre: Echo OBD driver; http://www.lustre.org/ [12647.113559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12647.136271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12648.483592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12648.572483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12648.736469] Lustre: lustre-MDT0000: new disk, initializing [12648.859619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12648.892703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12653.512071] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12664.958444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12665.065436] Lustre: 319315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12665.122143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12665.125498] Lustre: Skipped 1 previous similar message [12665.203354] Lustre: lustre-MDT0001: new disk, initializing [12665.300415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12665.387835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12665.396085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12671.626768] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12683.909977] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12691.277976] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12699.778895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12700.073525] Lustre: lustre-OST0000: new disk, initializing [12700.076131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12700.081856] Lustre: 321285:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12700.133975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12707.824585] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12709.573954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12709.597223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12709.761762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12720.111869] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12722.746682] Lustre: Mounted lustre-client [12723.876754] Lustre: Unmounted lustre-client [12738.061280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12738.066818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12738.073617] Lustre: Skipped 1 previous similar message [12738.086243] Lustre: Skipped 1 previous similar message [12742.554495] Lustre: server umount lustre-OST0000 complete [12748.256653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12748.279558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12748.281919] Lustre: Skipped 1 previous similar message [12748.293122] Lustre: Skipped 1 previous similar message [12753.376529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12753.402106] Lustre: Skipped 1 previous similar message [12758.497515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12758.506891] Lustre: Skipped 1 previous similar message [12762.079128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12762.518238] Lustre: server umount lustre-MDT0000 complete [12767.243558] LustreError: 322696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087327 with bad export cookie 13065269714900656255 [12767.245948] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12767.259916] LustreError: 322696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12767.768983] Lustre: server umount lustre-MDT0001 complete [12778.402460] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [12783.169039] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12787.445934] Key type lgssc unregistered [12787.922857] LNet: 323450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12787.929061] LNetError: 323450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12854.502245] LNet: Removed LNI 192.168.201.106@tcp [12855.907313] Key type .llcrypt unregistered [12855.909694] Key type ._llcrypt unregistered [12878.782554] Key type ._llcrypt registered [12878.788394] Key type .llcrypt registered [12878.959387] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [12895.963166] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12897.293516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12897.561856] alg: No test for adler32 (adler32-zlib) [12898.741906] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [12899.028174] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12900.743216] Key type lgssc registered [12902.237117] Lustre: Echo OBD driver; http://www.lustre.org/ [12937.771532] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:24:55 (1786087495) [12945.972576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12946.001122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12947.298386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12947.342601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12947.411432] Lustre: lustre-MDT0000: new disk, initializing [12947.483926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12947.494199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12952.707281] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12966.043406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12966.198469] Lustre: 326673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12966.257491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12966.262244] Lustre: Skipped 1 previous similar message [12966.357390] Lustre: lustre-MDT0001: new disk, initializing [12966.465356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12966.514539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12966.553579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12971.159204] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12983.280703] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12990.280245] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12997.643672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12997.867250] Lustre: lustre-OST0000: new disk, initializing [12997.870541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12997.876365] Lustre: 328646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12997.974438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13005.321433] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13006.466174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13006.483637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13006.726373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13017.500299] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13037.539158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13037.539846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13037.549760] LustreError: Skipped 1 previous similar message [13037.550626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13037.596273] Lustre: Skipped 1 previous similar message [13041.280807] Lustre: server umount lustre-OST0000 complete [13047.777140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13047.790618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13047.798543] Lustre: Skipped 2 previous similar messages [13052.897628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13052.909831] Lustre: Skipped 1 previous similar message [13057.010321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13057.036977] Lustre: Skipped 1 previous similar message [13060.063153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13060.453355] Lustre: server umount lustre-MDT0000 complete [13064.636824] LustreError: 327480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087624 with bad export cookie 6734317247498546098 [13064.641437] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13064.644947] LustreError: 327480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13065.237419] Lustre: server umount lustre-MDT0001 complete [13078.912409] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13081.837216] Key type lgssc unregistered [13082.204643] LNet: 330556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13082.210825] LNetError: 330556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13082.240157] LNet: Removed LNI 192.168.201.106@tcp [13083.412282] Key type .llcrypt unregistered [13083.416871] Key type ._llcrypt unregistered [13106.728420] Key type ._llcrypt registered [13106.730974] Key type .llcrypt registered [13106.893353] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13108.475485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13108.576445] alg: No test for adler32 (adler32-zlib) [13109.853930] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13110.259373] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13112.007752] Key type lgssc registered [13113.258645] Lustre: Echo OBD driver; http://www.lustre.org/ [13121.318745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13121.346069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13122.857609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13127.838571] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13132.257981] LustreError: 331832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13132.300872] LustreError: 331832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13137.388486] LustreError: 331831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.694587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13138.067779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13142.839814] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13150.864964] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13157.677439] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13164.365527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13164.608478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13171.458648] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13178.756273] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13194.064390] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13195.495758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13202.068353] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [13203.956446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13203.962415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13204.010051] Lustre: Skipped 2 previous similar messages [13204.022540] Lustre: Skipped 2 previous similar messages [13207.165622] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [13212.285915] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [13212.289512] Lustre: Skipped 3 previous similar messages [13214.179285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13214.430845] Lustre: server umount lustre-MDT0000 complete [13217.394112] LustreError: 331826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13218.920726] LustreError: 331811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087779 with bad export cookie 1362365513770675303 [13218.935585] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13218.938033] LustreError: 331811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13219.296482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13219.307495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13219.314353] Lustre: Skipped 3 previous similar messages [13222.508224] LustreError: 331827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13222.517921] LustreError: 331827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13225.369468] Lustre: server umount lustre-MDT0001 complete [13229.673392] Lustre: server umount lustre-OST0000 complete [13237.949518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13242.253619] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13244.927928] Lustre: 336014:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.106@tcp [13247.302408] Lustre: server umount MGS complete [13255.687683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13256.256433] LustreError: 336586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13256.279538] LustreError: 336586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13256.366549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13260.760974] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13265.903172] LustreError: 336586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13265.926026] LustreError: 336586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13269.725927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13270.431508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13275.108295] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13283.532620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13283.803419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13290.851579] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13298.343928] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13300.006850] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13305.904102] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13307.884262] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13315.144882] Lustre: server umount lustre-OST0000 complete [13321.700104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13321.713047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13321.716215] Lustre: Skipped 1 previous similar message [13321.716918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13321.767748] Lustre: Skipped 3 previous similar messages [13330.166117] Lustre: server umount lustre-MDT0000 complete [13331.871178] Lustre: 331141:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786087875/real 1786087875] req@000000007f01213f x1872848664720640/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786087891 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13331.883063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13331.889318] LustreError: 337314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13331.909532] LustreError: 337314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13332.895152] Lustre: 331143:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786087876/real 1786087876] req@00000000a7fc41ad x1872848664721408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786087892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13335.110567] LustreError: 338042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087895 with bad export cookie 1362365513770676143 [13335.121962] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13335.517742] Lustre: server umount lustre-MDT0001 complete [13348.152204] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13351.168538] Key type lgssc unregistered [13351.532839] LNet: 340022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13351.547824] LNetError: 340022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13351.582197] LNet: Removed LNI 192.168.201.106@tcp [13352.713330] Key type .llcrypt unregistered [13352.714767] Key type ._llcrypt unregistered [13371.670375] Key type ._llcrypt registered [13371.672390] Key type .llcrypt registered [13371.776755] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13386.355566] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13387.653960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13387.732873] alg: No test for adler32 (adler32-zlib) [13388.695195] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13388.890849] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13390.551342] Key type lgssc registered [13391.723300] Lustre: Echo OBD driver; http://www.lustre.org/ [13429.201776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13429.228946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13430.558339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13430.583238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13430.651315] Lustre: lustre-MDT0000: new disk, initializing [13430.813051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13430.839577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13436.187821] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13447.937040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13448.079921] Lustre: 343181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13448.161567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13448.170263] Lustre: Skipped 1 previous similar message [13448.297432] Lustre: lustre-MDT0001: new disk, initializing [13448.379788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13448.430070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13448.442719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13454.383243] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13463.481618] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13468.674940] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13475.730720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13476.059556] Lustre: lustre-OST0000: new disk, initializing [13476.066445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13476.072633] Lustre: 345149:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13476.185811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13482.617313] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13483.094704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13483.111254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13483.274170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13491.924874] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13497.223133] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13497.453484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13501.895397] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13502.076677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13503.972260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13503.983310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13503.994254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13503.998148] Lustre: Skipped 1 previous similar message [13504.010514] Lustre: Skipped 1 previous similar message [13509.109120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13509.120581] Lustre: Skipped 1 previous similar message [13514.213649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13514.221919] Lustre: Skipped 1 previous similar message [13518.305151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13518.643301] Lustre: server umount lustre-OST0000 complete [13524.449127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13524.465128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13524.471242] Lustre: Skipped 1 previous similar message [13529.678098] Lustre: server umount lustre-MDT0000 complete [13534.185335] LustreError: 343172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786088094 with bad export cookie 11231936884548845700 [13534.200330] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13534.621586] Lustre: server umount lustre-MDT0001 complete [13545.755262] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:35:03 (1786088103) [13562.348561] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13573.508720] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13616.998486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13617.374143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13617.418371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13617.608709] Lustre: lustre-MDT0000: new disk, initializing [13617.761601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13617.817676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13622.892978] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13634.208437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13634.279732] Lustre: 349913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13634.320919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13634.323250] Lustre: Skipped 1 previous similar message [13634.391455] Lustre: lustre-MDT0001: new disk, initializing [13634.488318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13634.557148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13634.568232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13640.465849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13650.645504] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13657.472787] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13665.743754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13666.068846] Lustre: lustre-OST0000: new disk, initializing [13666.072276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13666.084755] Lustre: 351882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13666.183223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13674.012592] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13675.647357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13675.661544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13675.866690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13686.399852] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13727.201201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13727.205875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13727.212742] LustreError: Skipped 2 previous similar messages [13727.238163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13727.243679] Lustre: Skipped 2 previous similar messages [13727.266356] Lustre: Skipped 1 previous similar message [13731.860387] Lustre: server umount lustre-OST0000 complete [13736.932576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13736.949868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13737.008450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13742.996299] Lustre: server umount lustre-MDT0000 complete [13747.683146] LustreError: 350914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13747.734180] LustreError: 350914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13747.895918] LustreError: 349905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786088308 with bad export cookie 11231936884548846582 [13747.905133] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13747.915784] LustreError: 349905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13748.307121] Lustre: server umount lustre-MDT0001 complete [13762.411638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13766.182223] Key type lgssc unregistered [13766.428876] LNet: 354057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13766.432858] LNetError: 354057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13767.464869] LNet: Removed LNI 192.168.201.106@tcp [13768.850979] Key type .llcrypt unregistered [13768.855801] Key type ._llcrypt unregistered [13883.129514] Key type ._llcrypt registered [13883.137880] Key type .llcrypt registered [13883.273143] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13883.647518] Key type .llcrypt unregistered [13883.653739] Key type ._llcrypt unregistered [13968.168846] Key type ._llcrypt registered [13968.173765] Key type .llcrypt registered [13968.312239] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13968.728616] Key type .llcrypt unregistered [13968.734266] Key type ._llcrypt unregistered [14003.143246] Key type ._llcrypt registered [14003.145806] Key type .llcrypt registered [14003.278630] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14003.627448] Key type .llcrypt unregistered [14003.629941] Key type ._llcrypt unregistered [14039.904305] Key type ._llcrypt registered [14039.911462] Key type .llcrypt registered [14040.125728] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14040.502629] Key type .llcrypt unregistered [14040.508670] Key type ._llcrypt unregistered [14122.470769] Key type ._llcrypt registered [14122.475359] Key type .llcrypt registered [14122.632347] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14123.000655] Key type .llcrypt unregistered [14123.004712] Key type ._llcrypt unregistered [14153.913638] Key type ._llcrypt registered [14153.917144] Key type .llcrypt registered [14154.078477] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14154.437790] Key type .llcrypt unregistered [14154.440990] Key type ._llcrypt unregistered [14186.254454] Key type ._llcrypt registered [14186.257435] Key type .llcrypt registered [14186.406697] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14186.734449] Key type .llcrypt unregistered [14186.736102] Key type ._llcrypt unregistered [14220.624826] Key type ._llcrypt registered [14220.626248] Key type .llcrypt registered [14220.837078] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14221.252317] Key type .llcrypt unregistered [14221.261559] Key type ._llcrypt unregistered [14253.244697] Key type ._llcrypt registered [14253.247876] Key type .llcrypt registered [14253.350860] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14253.613499] Key type .llcrypt unregistered [14253.616429] Key type ._llcrypt unregistered [14289.385435] Key type ._llcrypt registered [14289.394621] Key type .llcrypt registered [14289.622487] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14290.017320] Key type .llcrypt unregistered [14290.020818] Key type ._llcrypt unregistered [14363.110703] Key type ._llcrypt registered [14363.113799] Key type .llcrypt registered [14363.265543] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14363.679859] Key type .llcrypt unregistered [14363.684730] Key type ._llcrypt unregistered [14398.261829] Key type ._llcrypt registered [14398.263279] Key type .llcrypt registered [14398.492454] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14398.929512] Key type .llcrypt unregistered [14398.932986] Key type ._llcrypt unregistered [14433.366174] Key type ._llcrypt registered [14433.368995] Key type .llcrypt registered [14433.517537] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14433.815361] Key type .llcrypt unregistered [14433.818503] Key type ._llcrypt unregistered [14490.604631] Key type ._llcrypt registered [14490.607520] Key type .llcrypt registered [14490.731472] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [14507.832463] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [14509.008941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14509.244746] alg: No test for adler32 (adler32-zlib) [14510.445387] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [14510.692190] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [14512.487390] Key type lgssc registered [14513.642846] Lustre: Echo OBD driver; http://www.lustre.org/ [14560.131937] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14560.170147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14561.573801] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14561.642368] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14561.767099] Lustre: tst250fs-MDT0000: new disk, initializing [14561.854989] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14561.870833] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14567.264871] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14579.613038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14579.759240] Lustre: 374408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14579.801634] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14579.805983] Lustre: Skipped 1 previous similar message [14579.889732] Lustre: tst250fs-MDT0001: new disk, initializing [14579.977885] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14580.007478] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14580.023815] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14585.338987] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14590.856995] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14597.331717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14597.653917] Lustre: tst250fs-OST0000: new disk, initializing [14597.658951] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14597.665515] Lustre: 376037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14597.788201] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14599.589058] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14599.600714] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14599.794538] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14603.627327] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14614.046041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14614.291945] Lustre: tst250fs-OST0001: new disk, initializing [14614.300072] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14614.308246] Lustre: 376910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14614.441424] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14622.535733] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14624.457195] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14624.478107] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14624.551841] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14629.077331] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14632.066956] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14632.070886] Lustre: Skipped 2 previous similar messages [14655.460986] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14655.472728] Lustre: Skipped 1 previous similar message [14655.480419] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14655.487568] Lustre: Skipped 3 previous similar messages [14658.766707] Lustre: server umount tst250fs-MDT0000 complete [14662.624614] LustreError: 374401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786089222 with bad export cookie 461943066607489166 [14662.642605] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14662.645596] LustreError: 374401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14663.064121] Lustre: server umount tst250fs-MDT0001 complete [14677.842409] Lustre: server umount tst250fs-OST0000 complete [14681.058094] Lustre: 372628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786089225/real 1786089225] req@ffff9edfc9534700 x1872850132733568/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786089241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14681.091856] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14681.111089] Lustre: Skipped 2 previous similar messages [14682.148939] Lustre: server umount tst250fs-OST0001 complete [14702.416595] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [14711.081703] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [14748.047513] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:55:06 (1786089306) [14749.817820] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14766.197725] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [14776.644495] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [14815.107489] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14530 sec ========= 03:56:13 (1786089373) [14816.817920] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:56:15 (1786089375) === [14821.529220] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:56:19 (1786089379) === [14865.385077] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14868.857892] Key type lgssc unregistered [14869.257183] LNet: 386056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14869.265862] LNetError: 386056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14870.320869] LNet: Removed LNI 192.168.201.106@tcp [14871.511185] Key type .llcrypt unregistered [14871.512727] Key type ._llcrypt unregistered