[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479724813 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002425] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007604] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011037] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022688] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027034] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] *** VALIDATE tmpfs *** [ 0.031429] *** VALIDATE proc *** [ 0.033209] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035252] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.041702] debug: unmapping init [mem 0xffffffffb9259000-0xffffffffb9260fff] [ 0.044243] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045763] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046028] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053334] rcu: Hierarchical SRCU implementation. [ 0.055376] smp: Bringing up secondary CPUs ... [ 0.056619] x86: Booting SMP configuration: [ 0.057021] .... node #0, CPUs: #1 #2 #3 [ 0.061460] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.117215] node 0 deferred pages initialised in 50ms [ 0.121121] devtmpfs: initialized [ 0.122204] x86/mm: Memory block size: 128MB [ 0.124797] gcov: version magic: 0x41383552 [ 0.128239] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134312] pinctrl core: initialized pinctrl subsystem [ 0.137214] [ 0.137967] ************************************************************* [ 0.140015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143025] ** ** [ 0.145012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148015] ** ** [ 0.150014] ** This means that this kernel is built to expose internal ** [ 0.153014] ** IOMMU data structures, which may compromise security on ** [ 0.155022] ** your system. ** [ 0.158014] ** ** [ 0.161044] ** If you see this message and you are not debugging the ** [ 0.163017] ** kernel, report this immediately to your vendor! ** [ 0.166013] ** ** [ 0.168013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170021] ************************************************************* [ 0.174210] NET: Registered protocol family 16 [ 0.177536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189033] cpuidle: using governor menu [ 0.190748] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194664] PCI: Using configuration type 1 for base access [ 0.197161] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216093] cryptd: max_cpu_qlen set to 1000 [ 0.219139] ACPI: Added _OSI(Module Device) [ 0.220011] ACPI: Added _OSI(Processor Device) [ 0.222011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228376] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234642] ACPI: Interpreter enabled [ 0.236089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238023] ACPI: Using IOAPIC for interrupt routing [ 0.239122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243463] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254419] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266413] acpiphp: Slot [2] registered [ 0.268097] acpiphp: Slot [5] registered [ 0.269079] acpiphp: Slot [6] registered [ 0.271078] acpiphp: Slot [7] registered [ 0.273099] acpiphp: Slot [8] registered [ 0.275140] acpiphp: Slot [9] registered [ 0.277176] acpiphp: Slot [10] registered [ 0.278211] acpiphp: Slot [3] registered [ 0.280097] acpiphp: Slot [4] registered [ 0.281100] acpiphp: Slot [11] registered [ 0.283169] acpiphp: Slot [12] registered [ 0.284102] acpiphp: Slot [13] registered [ 0.286135] acpiphp: Slot [14] registered [ 0.288164] acpiphp: Slot [15] registered [ 0.290180] acpiphp: Slot [16] registered [ 0.291132] acpiphp: Slot [17] registered [ 0.293139] acpiphp: Slot [18] registered [ 0.295157] acpiphp: Slot [19] registered [ 0.297117] acpiphp: Slot [20] registered [ 0.298117] acpiphp: Slot [21] registered [ 0.299132] acpiphp: Slot [22] registered [ 0.301120] acpiphp: Slot [23] registered [ 0.303089] acpiphp: Slot [24] registered [ 0.304139] acpiphp: Slot [25] registered [ 0.306066] acpiphp: Slot [26] registered [ 0.307122] acpiphp: Slot [27] registered [ 0.309093] acpiphp: Slot [28] registered [ 0.310111] acpiphp: Slot [29] registered [ 0.312089] acpiphp: Slot [30] registered [ 0.313102] acpiphp: Slot [31] registered [ 0.315085] PCI host bridge to bus 0000:00 [ 0.317021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.323021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.328023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333122] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.337192] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340266] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.360017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365317] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369974] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.371037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.374831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407644] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.417015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455296] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.464013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504887] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518030] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.538014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.551014] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.560016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.571018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.593025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604747] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.619022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.627024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.656323] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.665024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.674024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.698020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.713689] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.716327] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.718496] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.721478] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.723271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.727031] iommu: Default domain type: Passthrough [ 0.729489] SCSI subsystem initialized [ 0.731197] ACPI: bus type USB registered [ 0.732147] usbcore: registered new interface driver usbfs [ 0.736137] usbcore: registered new interface driver hub [ 0.738110] usbcore: registered new device driver usb [ 0.740223] pps_core: LinuxPPS API ver. 1 registered [ 0.742017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.746088] PTP clock support registered [ 0.748179] EDAC MC: Ver: 3.0.0 [ 0.751224] PCI: Using ACPI for IRQ routing [ 0.753111] NetLabel: Initializing [ 0.754017] NetLabel: domain hash size = 128 [ 0.755011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.756138] NetLabel: unlabeled traffic allowed by default [ 0.759133] vgaarb: loaded [ 0.760600] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.763013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.771366] clocksource: Switched to clocksource kvm-clock [ 0.876601] VFS: Disk quotas dquot_6.6.0 [ 0.878064] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.880054] *** VALIDATE ramfs *** [ 0.881469] *** VALIDATE hugetlbfs *** [ 0.882918] pnp: PnP ACPI init [ 0.885110] pnp: PnP ACPI: found 6 devices [ 0.909674] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.913537] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.916071] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.918661] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.921443] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.924216] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.927452] NET: Registered protocol family 2 [ 0.930232] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.935692] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.939424] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.945306] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.949242] TCP: Hash tables configured (established 65536 bind 65536) [ 0.952347] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.955979] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959346] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963647] NET: Registered protocol family 1 [ 0.967537] RPC: Registered named UNIX socket transport module. [ 0.970571] RPC: Registered udp transport module. [ 0.972967] RPC: Registered tcp transport module. [ 0.975241] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.978323] NET: Registered protocol family 44 [ 0.980552] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.983110] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.985739] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.988792] PCI: CLS 0 bytes, default 64 [ 0.991001] Unpacking initramfs... [ 2.442302] debug: unmapping init [mem 0xffff8a437cc54000-0xffff8a437ffbffff] [ 2.446766] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.449098] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.452230] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.944453] Initialise system trusted keyrings [ 2.946473] Key type blacklist registered [ 2.949151] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.958042] zbud: loaded [ 2.960956] *** VALIDATE nfs *** [ 2.963828] *** VALIDATE nfs4 *** [ 2.965457] pstore: using deflate compression [ 2.968522] Platform Keyring initialized [ 3.095699] NET: Registered protocol family 38 [ 3.096887] Key type asymmetric registered [ 3.098297] Asymmetric key parser 'x509' registered [ 3.100043] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.103237] io scheduler mq-deadline registered [ 3.105217] io scheduler kyber registered [ 3.108557] io scheduler bfq registered [ 3.111140] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.114861] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.118368] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.121677] ACPI: Power Button [PWRF] [ 3.129155] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.136802] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.154802] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.163119] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.183574] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.211477] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.240397] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.245296] Non-volatile memory driver v1.3 [ 3.247577] Linux agpgart interface v0.103 [ 3.295756] virtio_blk virtio1: [vda] 136576 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.300021] vda: detected capacity change from 0 to 69926912 [ 3.316809] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.319760] vdb: detected capacity change from 0 to 1073741824 [ 3.338744] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.341590] vdc: detected capacity change from 0 to 2621440000 [ 3.356851] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.360405] vdd: detected capacity change from 0 to 2621440000 [ 3.375763] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379857] vde: detected capacity change from 0 to 4294967296 [ 3.397094] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.400281] vdf: detected capacity change from 0 to 4294967296 [ 3.411337] libphy: Fixed MDIO Bus: probed [ 3.417245] usbcore: registered new interface driver usbserial_generic [ 3.419796] usbserial: USB Serial support registered for generic [ 3.422257] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.426403] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.428312] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.430629] mousedev: PS/2 mouse device common for all mice [ 3.434651] rtc_cmos 00:05: RTC can wake from S4 [ 3.436779] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.440617] rtc_cmos 00:05: registered as rtc0 [ 3.442958] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.443980] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.446639] intel_pstate: CPU model not supported [ 3.451735] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.455872] hid: raw HID events driver (C) Jiri Kosina [ 3.458764] usbcore: registered new interface driver usbhid [ 3.461077] usbhid: USB HID core driver [ 3.462665] drop_monitor: Initializing network drop monitor service [ 3.465469] Initializing XFRM netlink socket [ 3.467555] NET: Registered protocol family 10 [ 3.473758] Segment Routing with IPv6 [ 3.475604] NET: Registered protocol family 17 [ 3.478386] mpls_gso: MPLS GSO support [ 3.485476] RAS: Correctable Errors collector initialized. [ 3.488324] AVX version of gcm_enc/dec engaged. [ 3.490227] AES CTR mode by8 optimization enabled [ 3.577201] sched_clock: Marking stable (3577040761, 0)->(4474769610, -897728849) [ 3.581513] registered taskstats version 1 [ 3.583538] Loading compiled-in X.509 certificates [ 3.587253] zswap: loaded using pool lzo/zbud [ 3.611774] Key type big_key registered [ 3.625750] Key type encrypted registered [ 3.628116] ima: No TPM chip found, activating TPM-bypass! [ 3.630738] ima: Allocated hash algorithm: sha1 [ 3.632881] ima: No architecture policies found [ 3.635161] evm: Initialising EVM extended attributes: [ 3.637205] evm: security.selinux [ 3.638651] evm: security.ima [ 3.640000] evm: security.capability [ 3.641109] evm: HMAC attrs: 0x1 [ 3.643592] rtc_cmos 00:05: setting system clock to 2026-06-12 04:25:10 UTC (1781238310) [ 3.650821] debug: unmapping init [mem 0xffffffffba203000-0xffffffffba3fffff] [ 3.653677] debug: unmapping init [mem 0xffffffffb8f82000-0xffffffffb9258fff] [ 3.664249] Write protecting the kernel read-only data: 28672k [ 3.668249] debug: unmapping init [mem 0xffffffffb7603000-0xffffffffb77fffff] [ 3.671835] debug: unmapping init [mem 0xffffffffb7f14000-0xffffffffb7ffffff] [ 3.710252] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.721862] systemd[1]: Detected virtualization kvm. [ 3.724053] systemd[1]: Detected architecture x86-64. [ 3.726021] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.760792] systemd[1]: No hostname configured. [ 3.763169] systemd[1]: Set hostname to . [ 3.765764] random: systemd: uninitialized urandom read (16 bytes read) [ 3.768896] systemd[1]: Initializing machine ID from random generator. [ 3.833553] random: ln: uninitialized urandom read (6 bytes read) [ 3.928352] random: systemd: uninitialized urandom read (16 bytes read) [ 3.931334] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.935609] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.939761] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.584741] device-mapper: uevent: version 1.0.3 [ 4.587500] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.305448] random: fast init done [ 5.317518] virtio_net virtio0 ens2: renamed from eth0 [ 5.417179] scsi host0: ata_piix [ 5.421949] scsi host1: ata_piix [ 5.423712] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.426969] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.712078] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.153509] random: crng init done [ 10.166883] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.462728] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.254077] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.067435] SELinux: Disabled at runtime. [ 15.179750] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.205383] systemd[1]: Detected virtualization kvm. [ 15.213750] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.482985] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.491168] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.498423] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.503583] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.511044] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.537638] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.550531] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [[ 17.056928] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.825624] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.426704] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.537290] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.975859] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.044816] EDAC sbridge: Ver: 1.1.2 [ 23.015082] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 23.626288] NFS: Registering the id_resolver key type [ 23.629534] Key type id_resolver registered [ 23.631979] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 44.922097] hrtimer: interrupt took 3145915 ns [ 79.128640] libcfs: loading out-of-tree module taints kernel. [ 79.170193] Key type ._llcrypt registered [ 79.171991] Key type .llcrypt registered [ 79.281918] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 96.982277] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 98.665521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.676320] alg: No test for adler32 (adler32-zlib) [ 100.073625] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 100.997439] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 102.728542] Key type lgssc registered [ 104.603422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.127599] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.377303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 173.496627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 173.548562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.995089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 175.020432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 175.106466] Lustre: lustre-MDT0000: new disk, initializing [ 175.176341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.202537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.737705] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.826670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.949884] Lustre: 6505:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.981188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.989471] Lustre: Skipped 1 previous similar message [ 196.136487] Lustre: lustre-MDT0001: new disk, initializing [ 196.228897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.307877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.318731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.739908] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.961524] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.473789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.825600] Lustre: lustre-OST0000: new disk, initializing [ 215.831692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.839911] Lustre: 8408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.937760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.757936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.774271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.908561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.091490] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.944417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.080885] Lustre: lustre-OST0001: new disk, initializing [ 238.085223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 238.092806] Lustre: 9465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.206243] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.520616] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.797304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.810586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.880276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.718909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 264.078847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.662939] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.656766] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 283.026832] Lustre: DEBUG MARKER: Client: 2.17.53.78 [ 285.864740] Lustre: DEBUG MARKER: MDS: 2.17.53.78 [ 288.623424] Lustre: DEBUG MARKER: OSS: 2.17.53.78 [ 290.455254] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 00:29:55 EDT 2026 [ 308.644900] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 310.438274] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 327.654938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.660173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.674134] Lustre: Skipped 1 previous similar message [ 332.396024] Lustre: server umount lustre-MDT0000 complete [ 335.986123] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238642 with bad export cookie 1892384897922169436 [ 335.992611] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.997143] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 337.889065] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.890585] LustreError: 8416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.891035] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.891039] Lustre: Skipped 3 previous similar messages [ 337.896854] Lustre: Skipped 3 previous similar messages [ 337.945923] LustreError: 8416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 340.897106] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.898701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.919311] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 340.939388] Lustre: Skipped 2 previous similar messages [ 342.498166] Lustre: server umount lustre-MDT0001 complete [ 346.236482] Lustre: server umount lustre-OST0000 complete [ 349.912125] Lustre: server umount lustre-OST0001 complete [ 356.291479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 365.680947] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 401.335623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.640906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.679213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.757268] Lustre: lustre-MDT0000: new disk, initializing [ 401.833195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.856947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.799940] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 416.977323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.111898] Lustre: 14979:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.176937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.181478] Lustre: Skipped 1 previous similar message [ 417.238744] Lustre: lustre-MDT0001: new disk, initializing [ 417.328550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.389566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.407239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 421.818463] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 429.129564] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 432.574538] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 439.213739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 439.646795] Lustre: lustre-OST0000: new disk, initializing [ 439.651333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 439.664990] Lustre: 16723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 439.809513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.929709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 440.955178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 441.107250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 445.942108] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 453.733865] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 458.560289] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 458.782987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 463.217216] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 463.490983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 466.916725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.923962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.929995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.932542] Lustre: Skipped 1 previous similar message [ 466.942837] Lustre: Skipped 1 previous similar message [ 471.388334] Lustre: server umount lustre-OST0000 complete [ 477.165423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.175228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.189604] Lustre: Skipped 1 previous similar message [ 477.204543] Lustre: Skipped 1 previous similar message [ 481.567517] Lustre: server umount lustre-MDT0000 complete [ 482.277756] LustreError: 16443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.297368] LustreError: 16443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 485.476941] LustreError: 15767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238792 with bad export cookie 1892384897922170969 [ 485.481555] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.495058] LustreError: 15767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 485.816551] Lustre: server umount lustre-MDT0001 complete [ 491.987811] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:33:17 (1781238797) [ 509.891511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.338508] LustreError: 19452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.401322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.348119] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 515.555637] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.502459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.934617] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 532.479980] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 536.430405] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 538.279632] Lustre: Failing over lustre-MDT0000 [ 538.490969] Lustre: server umount lustre-MDT0000 complete [ 541.155800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.172777] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.190796] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 557.121311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.310238] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.458931] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 562.660058] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 562.663858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 562.702368] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 569.540669] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 578.025356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 578.032023] LustreError: Skipped 1 previous similar message [ 578.034824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.041130] Lustre: Skipped 1 previous similar message [ 578.042403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.256090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 588.822841] Lustre: server umount lustre-MDT0000 complete [ 592.356243] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.381940] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 592.630915] LustreError: 20793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238899 with bad export cookie 1892384897922172383 [ 592.639939] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.654634] LustreError: 20793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 593.037741] Lustre: server umount lustre-MDT0001 complete [ 601.258305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.728890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.740499] Lustre: Skipped 2 previous similar messages [ 606.338885] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 615.017093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.826338] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 626.091301] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 630.804305] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 633.123123] Lustre: Failing over lustre-MDT0000 [ 633.593302] Lustre: server umount lustre-MDT0000 complete [ 635.876270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.883187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.896705] LustreError: 22448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.911172] LustreError: 22448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 652.260426] Lustre: 3653:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781238943/real 1781238943] req@ffff8a43ffd952c0 x1867763844045824/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781238959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 652.309456] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.007840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.501047] LustreError: 3652:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4400868b40 x1867763844048640/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 662.838743] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 667.449436] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 668.129680] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 668.166442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 668.178666] Lustre: Skipped 1 previous similar message [ 668.198732] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 675.187697] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 682.474435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.480477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 682.486405] Lustre: Skipped 2 previous similar messages [ 682.498177] Lustre: Skipped 6 previous similar messages [ 693.217937] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 693.382662] Lustre: server umount lustre-MDT0000 complete [ 697.180426] LustreError: 22430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239004 with bad export cookie 1892384897922173321 [ 697.194148] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.205661] LustreError: 22430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 697.498298] Lustre: server umount lustre-MDT0001 complete [ 705.748668] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:36:51 (1781239011) [ 719.976708] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 729.989542] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 768.048261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 778.911031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.168310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 779.220611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 779.383214] Lustre: lustre-MDT0000: new disk, initializing [ 779.550647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.556744] Lustre: Skipped 2 previous similar messages [ 779.569957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 784.200435] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 794.622764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.765196] Lustre: 28432:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 795.086212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 798.920800] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 805.645654] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 809.101489] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 814.776061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 814.974453] Lustre: 30174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 816.117673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 816.165262] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 816.211627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 816.223713] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 819.599500] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 826.147705] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.592702] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 851.937455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.944459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.948398] LustreError: Skipped 1 previous similar message [ 851.949272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 851.962581] Lustre: Skipped 1 previous similar message [ 851.982053] Lustre: Skipped 5 previous similar messages [ 855.744422] Lustre: server umount lustre-OST0000 complete [ 861.668154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 865.686959] Lustre: server umount lustre-MDT0000 complete [ 867.306154] LustreError: 29225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.321631] LustreError: 29225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 868.977182] LustreError: 30998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239175 with bad export cookie 1892384897922173727 [ 868.981076] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 868.993227] LustreError: 30998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 869.480291] Lustre: server umount lustre-MDT0001 complete [ 878.269254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 880.600428] Key type lgssc unregistered [ 880.845876] LNet: 32021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.850840] LNetError: 32021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.861530] LNet: Removed LNI 192.168.203.123@tcp [ 881.540271] Key type .llcrypt unregistered [ 881.544062] Key type ._llcrypt unregistered [ 909.859277] Key type ._llcrypt registered [ 909.862863] Key type .llcrypt registered [ 910.017849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 910.842560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.875253] alg: No test for adler32 (adler32-zlib) [ 912.238483] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 912.518791] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 914.272161] Key type lgssc registered [ 915.463117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.312862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.351034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.939289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.614931] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 935.406331] LustreError: 33421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.430947] LustreError: 33421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 938.761123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.140906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 943.316977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 948.939201] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 952.997749] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 959.604465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.069850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.203895] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 966.160933] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 970.391981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 972.181528] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 987.127531] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:41:32 (1781239292) [ 989.432968] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 991.373994] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 999.014538] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:41:43 (1781239303) [ 1007.928223] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:41:52 (1781239312) [ 1008.955608] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1017.347832] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1017.354672] Lustre: Skipped 2 previous similar messages [ 1019.316224] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:42:04 (1781239324) [ 1022.859897] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1022.867509] Lustre: Skipped 3 previous similar messages [ 1041.076177] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1041.092834] Lustre: Skipped 6 previous similar messages [ 1043.925555] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:42:28 (1781239348) [ 1047.100375] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1051.478264] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1060.581468] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:42:45 (1781239365) [ 1062.573446] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1062.576930] Lustre: Skipped 3 previous similar messages [ 1077.173312] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:43:01 (1781239381) [ 1093.606187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.619813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.653681] Lustre: Skipped 1 previous similar message [ 1093.656452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.729704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.748386] Lustre: Skipped 2 previous similar messages [ 1099.044249] Lustre: server umount lustre-OST0000 complete [ 1103.841312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.848417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.852970] Lustre: Skipped 1 previous similar message [ 1103.865228] Lustre: Skipped 1 previous similar message [ 1108.962972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.968208] Lustre: Skipped 1 previous similar message [ 1109.524448] Lustre: server umount lustre-MDT0000 complete [ 1113.788786] LustreError: 33403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239420 with bad export cookie 4693927898195244382 [ 1113.793076] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1113.799443] LustreError: 33403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1114.187516] Lustre: server umount lustre-MDT0001 complete [ 1126.107247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1128.871565] Key type lgssc unregistered [ 1129.226216] LNet: 40921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.231707] LNetError: 40921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.252739] LNet: Removed LNI 192.168.203.123@tcp [ 1130.127488] Key type .llcrypt unregistered [ 1130.129996] Key type ._llcrypt unregistered [ 1151.252208] Key type ._llcrypt registered [ 1151.254015] Key type .llcrypt registered [ 1151.359721] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1167.325481] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1168.765653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1168.914865] alg: No test for adler32 (adler32-zlib) [ 1170.220874] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1170.465801] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1172.153610] Key type lgssc registered [ 1173.393224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.682913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.716676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.000993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1213.033205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1213.119952] Lustre: lustre-MDT0000: new disk, initializing [ 1213.233499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.259974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1217.368567] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1227.831819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.975560] Lustre: 44041:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1228.059112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1228.069134] Lustre: Skipped 1 previous similar message [ 1228.233388] Lustre: lustre-MDT0001: new disk, initializing [ 1228.365399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.427803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1228.443906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1233.655130] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1241.854493] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1245.746383] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1252.466751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.817393] Lustre: lustre-OST0000: new disk, initializing [ 1252.823070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1252.832397] Lustre: 45782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1252.900863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.416213] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1259.631025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1259.652684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1259.798044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1266.340366] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1271.177912] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1271.350467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1276.000205] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1276.192114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1280.484333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1280.489693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.492689] LustreError: Skipped 1 previous similar message [ 1280.506063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1280.508716] Lustre: Skipped 1 previous similar message [ 1280.522392] Lustre: Skipped 1 previous similar message [ 1285.601823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.609532] Lustre: Skipped 1 previous similar message [ 1290.729371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.744626] Lustre: Skipped 1 previous similar message [ 1292.256467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1292.459811] Lustre: server umount lustre-OST0000 complete [ 1299.939174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1299.947043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.967742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.968211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.484149] Lustre: server umount lustre-MDT0000 complete [ 1306.428789] LustreError: 44035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239613 with bad export cookie 12831650159724513450 [ 1306.441733] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.447762] LustreError: 44035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1306.796319] Lustre: server umount lustre-MDT0001 complete [ 1317.482869] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:47:01 (1781239621) [ 1326.181871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.751054] LustreError: 48098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.767250] LustreError: 48098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1326.843142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.954647] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1332.196242] LustreError: 48097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.299410] LustreError: 48098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.863654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.132586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1345.891327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1352.604669] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1356.741016] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1364.297971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.751879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.633365] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1378.887580] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1385.015981] LustreError: 50289:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1386.045214] Lustre: Modifying parameter general.timeout=1 in log params [ 1387.070719] Lustre: Modifying parameter general.timeout=2 in log params [ 1389.876505] Lustre: Modifying parameter general.timeout=5 in log params [ 1389.885533] Lustre: Skipped 2 previous similar messages [ 1394.693667] Lustre: Modifying parameter general.timeout=10 in log params [ 1394.695910] Lustre: Skipped 4 previous similar messages [ 1403.073474] Lustre: Modifying parameter general.timeout=19 in log params [ 1403.080395] Lustre: Skipped 8 previous similar messages [ 1419.399762] Lustre: Modifying parameter general.timeout=35 in log params [ 1419.404532] Lustre: Skipped 15 previous similar messages [ 1451.422172] Lustre: Modifying parameter general.timeout=71 in log params [ 1451.425810] Lustre: Skipped 35 previous similar messages [ 1519.020516] Lustre: Disabling parameter general.timeout= in log params [ 1519.025829] Lustre: Skipped 58 previous similar messages [ 1523.033635] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:50:27 (1781239827) [ 1526.161782] Lustre: 57244:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1528.183800] Lustre: 57342:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1530.410939] Lustre: 57441:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1536.045248] Lustre: 57755:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1536.062862] Lustre: 57755:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1566.760813] Lustre: 59118:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1566.798168] LustreError: 59118:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1571.712469] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:51:16 (1781239876) [ 1586.658261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.666181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1586.681922] Lustre: Skipped 2 previous similar messages [ 1586.702304] Lustre: Skipped 4 previous similar messages [ 1592.678254] Lustre: server umount lustre-MDT0000 complete [ 1596.907203] LustreError: 48808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.940271] LustreError: 48808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1597.066804] LustreError: 48078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239903 with bad export cookie 12831650159724514332 [ 1597.076703] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.088506] LustreError: 48078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1597.523559] Lustre: server umount lustre-MDT0001 complete [ 1611.492487] Lustre: server umount lustre-OST0000 complete [ 1642.738824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.216578] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1643.538356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.775987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1657.980763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.190464] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1658.227988] Lustre: 61651:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1658.402230] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1658.430391] Lustre: Skipped 3 previous similar messages [ 1663.443558] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1669.891470] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1674.819076] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1682.803281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1682.994966] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1683.311824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.322322] Lustre: Skipped 1 previous similar message [ 1690.034438] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1695.949889] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1698.061835] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1698.069672] Lustre: Skipped 23 previous similar messages [ 1698.112343] Lustre: 63881:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1706.977290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1706.981877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.988212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1707.011288] Lustre: Skipped 1 previous similar message [ 1710.435467] Lustre: server umount lustre-OST0000 complete [ 1715.176647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1715.183557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.337055] LustreError: 61660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.363945] LustreError: 61660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1723.463597] Lustre: server umount lustre-MDT0000 complete [ 1727.613863] LustreError: 61644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240034 with bad export cookie 12831650159724517482 [ 1727.615633] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1727.632991] LustreError: 61644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1727.928451] Lustre: server umount lustre-MDT0001 complete [ 1741.576603] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1744.309599] Key type lgssc unregistered [ 1744.631983] LNet: 65073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.641705] LNetError: 65073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.668812] LNet: Removed LNI 192.168.203.123@tcp [ 1745.550909] Key type .llcrypt unregistered [ 1745.553877] Key type ._llcrypt unregistered [ 1772.477576] Key type ._llcrypt registered [ 1772.483235] Key type .llcrypt registered [ 1772.686369] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1773.909748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1774.131159] alg: No test for adler32 (adler32-zlib) [ 1775.349870] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1775.627491] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1777.353800] Key type lgssc registered [ 1778.678696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1786.350605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1786.375585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.995446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.110742] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1798.125881] LustreError: 66471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.138909] LustreError: 66471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1801.835406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.608119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.189615] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1813.428435] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1818.387186] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1826.030357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.750105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.720645] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1840.362547] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1848.335415] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1858.533888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1858.538628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.545570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.575575] Lustre: Skipped 2 previous similar messages [ 1863.651456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.663417] Lustre: Skipped 4 previous similar messages [ 1864.763116] Lustre: server umount lustre-MDT0000 complete [ 1868.770061] LustreError: 66470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.798814] LustreError: 66470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1868.852312] LustreError: 67195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240175 with bad export cookie 7311607250586608933 [ 1868.858566] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1868.863177] LustreError: 67195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1869.182390] Lustre: server umount lustre-MDT0001 complete [ 1883.742546] Lustre: server umount lustre-OST0000 complete [ 1911.904354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.383462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1912.443503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1912.817298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.462567] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1926.929162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.010244] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1927.060073] Lustre: 71002:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1927.489165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.319680] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1938.578694] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1943.781938] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1951.596295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1951.822082] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1952.231971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.480355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1966.143562] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1968.212095] Lustre: 73238:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1970.560629] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1977.830554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.833126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.869432] Lustre: Skipped 1 previous similar message [ 1977.875543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1982.961964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1982.975483] Lustre: Skipped 2 previous similar messages [ 1983.754317] Lustre: server umount lustre-OST0000 complete [ 1988.585124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1988.589580] LustreError: Skipped 1 previous similar message [ 1988.591595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.604963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.189367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.314407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.319776] Lustre: Skipped 3 previous similar messages [ 2002.400374] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2002.519628] Lustre: server umount lustre-MDT0000 complete [ 2003.427366] LustreError: 71009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.456182] LustreError: 71009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2007.093634] LustreError: 70994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240313 with bad export cookie 7311607250586609780 [ 2007.096703] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.111189] LustreError: 70994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2007.629447] Lustre: server umount lustre-MDT0001 complete [ 2020.426466] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2024.005577] Key type lgssc unregistered [ 2024.311518] LNet: 74442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.317734] LNetError: 74442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.433110] LNet: Removed LNI 192.168.203.123@tcp [ 2026.480495] Key type .llcrypt unregistered [ 2026.482716] Key type ._llcrypt unregistered [ 2054.956554] Key type ._llcrypt registered [ 2054.964644] Key type .llcrypt registered [ 2055.074792] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2056.174183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.236050] alg: No test for adler32 (adler32-zlib) [ 2057.507646] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2057.764715] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2059.432283] Key type lgssc registered [ 2060.755584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.074337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.103501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.798836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.876119] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2080.240489] LustreError: 75840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.260865] LustreError: 75840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2084.940548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2085.658055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.395743] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2096.160248] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2100.567534] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2108.171295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2108.684189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.480993] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2120.942291] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2128.303663] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2129.378874] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2130.453703] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2132.907129] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2132.911876] Lustre: Skipped 1 previous similar message [ 2137.166167] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2137.170702] Lustre: Skipped 3 previous similar messages [ 2145.177641] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2145.180589] Lustre: Skipped 7 previous similar messages [ 2161.632091] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2161.638229] Lustre: Skipped 16 previous similar messages [ 2194.687881] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2194.693471] Lustre: Skipped 33 previous similar messages [ 2236.483140] Lustre: 83619:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2239.876974] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:02:24 (1781240544) [ 2241.828179] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2243.840797] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:02:28 (1781240548) [ 2254.307886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2254.311655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.331032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2254.334137] LustreError: Skipped 1 previous similar message [ 2254.361121] Lustre: Skipped 1 previous similar message [ 2255.900216] Lustre: server umount lustre-OST0000 complete [ 2264.547987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.549176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.579327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.594373] Lustre: Skipped 2 previous similar messages [ 2266.766263] Lustre: server umount lustre-MDT0000 complete [ 2269.668507] LustreError: 75839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.692532] LustreError: 75839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2271.148814] LustreError: 75820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240578 with bad export cookie 757995065710451112 [ 2271.164877] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.168495] LustreError: 75820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2271.825389] Lustre: server umount lustre-MDT0001 complete [ 2284.389397] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2287.281932] Key type lgssc unregistered [ 2287.554526] LNet: 84852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.573229] LNetError: 84852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.610450] LNet: Removed LNI 192.168.203.123@tcp [ 2288.399771] Key type .llcrypt unregistered [ 2288.403672] Key type ._llcrypt unregistered [ 2293.082809] Key type ._llcrypt registered [ 2293.086253] Key type .llcrypt registered [ 2293.259522] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2298.029343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules [ 2299.936484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.974189] alg: No test for adler32 (adler32-zlib) [ 2301.188223] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2301.535421] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2303.256304] Key type lgssc registered [ 2304.950932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2306.279784] LustreError: 86053:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2322.422576] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2327.285537] LustreError: 86053:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2332.625927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2332.635968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.319532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.070371] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2344.423792] LustreError: 87027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.434201] LustreError: 87027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2349.541779] LustreError: 87047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.704544] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:04:14 (1781240654) [ 2354.664644] LustreError: 87027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.929471] LustreError: 87060:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2354.944905] LustreError: 87060:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2354.953641] LustreError: 87060:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2355.546829] Lustre: server umount lustre-MDT0000 complete [ 2375.689970] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2379.452779] Key type lgssc unregistered [ 2379.935990] LNet: 88521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.962567] LNetError: 88521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.994550] LNet: Removed LNI 192.168.203.123@tcp [ 2381.151405] Key type .llcrypt unregistered [ 2381.152876] Key type ._llcrypt unregistered [ 2400.238578] Key type ._llcrypt registered [ 2400.241683] Key type .llcrypt registered [ 2400.399816] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2401.683597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.799411] alg: No test for adler32 (adler32-zlib) [ 2402.853909] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2403.125143] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2404.832194] Key type lgssc registered [ 2406.017681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.664606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2413.698930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.411731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.261478] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2425.827653] LustreError: 89723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.850103] LustreError: 89723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2429.863594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.474948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2435.674288] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2441.428067] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2445.532452] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2452.908028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2453.724242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.520339] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2466.944448] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2519.127750] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:07:04 (1781240824) [ 2520.883530] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2522.956426] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:07:07 (1781240827) [ 2527.719989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.721284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2527.729047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.731812] Lustre: Skipped 2 previous similar messages [ 2532.835220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.844462] Lustre: Skipped 4 previous similar messages [ 2533.777576] Lustre: server umount lustre-MDT0000 complete [ 2537.801400] LustreError: 89703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240844 with bad export cookie 10440636093414498599 [ 2537.813189] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.820483] LustreError: 89703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2537.961114] LustreError: 92178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.962899] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.015476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2538.329504] Lustre: server umount lustre-MDT0001 complete [ 2545.505253] Lustre: server umount lustre-OST0000 complete [ 2557.939942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.433638] LustreError: 93680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.595943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.674338] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2564.069057] LustreError: 93681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.167898] LustreError: 93680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.864796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.747763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.657590] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2585.344947] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2590.408946] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2605.099888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.440249] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2605.446423] LustreError: 93668:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2605.474789] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2605.489483] LustreError: 95538:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2605.498643] LustreError: 95538:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2605.508229] LustreError: 95538:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2605.618600] Lustre: server umount lustre-OST0000 complete [ 2605.625935] LustreError: 95538:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2614.846650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2615.067583] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2615.077608] LustreError: 93670:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2615.086797] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2615.091916] LustreError: 96000:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2615.106114] LustreError: 96000:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2615.109140] LustreError: 96000:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2615.192464] Lustre: server umount lustre-OST0000 complete [ 2615.196619] LustreError: 96000:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2625.998542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.239995] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2626.248650] Lustre: Found index 0 for lustre-OST0000, updating log [ 2626.264756] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2626.377290] Lustre: lustre-OST0000: new disk, initializing [ 2626.386696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2626.396514] Lustre: 96534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2626.838582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.352274] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2638.274466] Lustre: Failing over lustre-OST0000 [ 2638.556553] Lustre: server umount lustre-OST0000 complete [ 2643.690348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.708195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.719816] Lustre: Skipped 1 previous similar message [ 2649.061175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.177902] LustreError: 94395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.203150] LustreError: 94395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2655.281678] Lustre: server umount lustre-MDT0000 complete [ 2659.763317] LustreError: 93660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240966 with bad export cookie 10440636093414501343 [ 2659.773188] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.791386] LustreError: 93660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2660.137262] Lustre: server umount lustre-MDT0001 complete [ 2670.943514] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:09:35 (1781240975) [ 2691.255470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2701.919607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.287073] LustreError: 98848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.303030] LustreError: 98848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2702.405573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.911548] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2717.026419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.102076] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2726.455595] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2734.441090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2735.254918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.260986] Lustre: Skipped 1 previous similar message [ 2735.284485] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2742.440419] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2742.463983] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2742.496852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2742.934352] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2747.639693] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2747.673041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2747.693073] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2747.698358] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2747.716128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2754.033718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.202766] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2754.315844] Lustre: lustre-OST0001: new disk, initializing [ 2754.320388] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2754.331308] Lustre: 101120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2762.289869] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2763.380118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2763.398829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2763.571652] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2771.389534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2784.614423] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2791.077414] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 2791.087257] Lustre: Skipped 1 previous similar message [ 2794.468360] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.486955] Lustre: Skipped 2 previous similar messages [ 2797.101667] Lustre: server umount lustre-MDT0001 complete [ 2799.599758] LustreError: 98844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.626137] LustreError: 98844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2810.144444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.297816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2810.303817] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2810.366912] Lustre: 98837:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2811.427037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2811.444933] Lustre: Skipped 1 previous similar message [ 2811.502693] LustreError: 102620:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2811.517312] LustreError: 102620:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2811.541124] LustreError: 98840:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2811.768289] LustreError: 102633:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.123@tcp: cfg command failed: rc = -17 [ 2811.780732] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2811.780732] [ 2816.498266] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2816.853261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2827.885292] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:12:13 (1781241133) [ 2848.182855] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2867.818487] Lustre: 104685:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2897.282413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2901.941696] Lustre: 105820:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3000.291613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3000.304311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.317488] Lustre: Skipped 1 previous similar message [ 3000.321504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.328740] Lustre: Skipped 4 previous similar messages [ 3005.331981] Lustre: server umount lustre-MDT0000 complete [ 3005.923219] LustreError: 106167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.946161] LustreError: 106167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3009.123583] LustreError: 98829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241315 with bad export cookie 10440636093414502274 [ 3009.123950] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3009.145692] LustreError: 98829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3009.504816] Lustre: server umount lustre-MDT0001 complete [ 3024.140029] Lustre: server umount lustre-OST0000 complete [ 3026.914656] Lustre: 89032:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781241317/real 1781241317] req@ffff8a43e11321c0 x1867766258050560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781241333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3026.948476] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.966353] Lustre: Skipped 3 previous similar messages [ 3028.647630] Lustre: server umount lustre-OST0001 complete [ 3034.693956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3045.472390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3054.917308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3067.665166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3096.702538] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3107.851666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.887043] Lustre: lustre-MDT0000: reset Object Index mappings [ 3108.380538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.683297] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3121.449525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.475785] Lustre: lustre-MDT0001: reset Object Index mappings [ 3125.950697] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3128.876789] Lustre: 110893:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3135.562127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.695476] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3146.546867] LustreError: 111247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.578942] LustreError: 111247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3146.579311] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3152.279675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.119607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3154.124149] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3154.250287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3159.700097] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3168.369482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3172.964425] Lustre: 112737:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3227.808844] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:18:52 (1781241532) [ 3255.265256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3255.279739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.302338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.307547] Lustre: Skipped 3 previous similar messages [ 3260.713916] Lustre: server umount lustre-MDT0000 complete [ 3264.108546] LustreError: 109763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241570 with bad export cookie 10440636093414549384 [ 3264.110572] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.149578] LustreError: 109763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3270.869487] Lustre: server umount lustre-MDT0001 complete [ 3281.557566] Lustre: server umount lustre-OST0000 complete [ 3291.991188] Lustre: server umount lustre-OST0001 complete [ 3298.787427] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3311.504945] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3348.454228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.761801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3348.799236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3348.931243] Lustre: lustre-MDT0000: new disk, initializing [ 3349.044138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3353.701363] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3361.180415] Lustre: server umount lustre-MDT0000 complete [ 3371.730394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.230223] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3372.249474] Lustre: 116951:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3372.288338] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3372.571773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.579681] Lustre: Skipped 4 previous similar messages [ 3378.994789] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3385.076023] Lustre: server umount lustre-MDT0000 complete [ 3395.881743] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:21:40 (1781241700) [ 3397.711201] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3400.027581] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:21:44 (1781241704) [ 3418.796116] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3431.632815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.242716] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3448.229053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.463625] Lustre: 118970:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3448.548041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3448.553136] Lustre: Skipped 1 previous similar message [ 3448.684105] Lustre: lustre-MDT0001: new disk, initializing [ 3448.886286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3448.907112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3453.701288] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3458.916471] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3465.616628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3465.876626] Lustre: lustre-OST0000: new disk, initializing [ 3465.890953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3465.901927] Lustre: 120512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3467.126808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3467.148580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3467.306470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3471.857396] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3482.494946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.734076] Lustre: lustre-OST0001: new disk, initializing [ 3482.740215] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3482.754175] Lustre: 121371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3484.330812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3484.349446] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3484.461080] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3488.649596] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3497.820637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3500.860404] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3904.325691] Lustre: 127767:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3904.333876] Lustre: 127767:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3904.340943] Lustre: 127767:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3904.354274] Lustre: 127767:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3904.364889] Lustre: 127767:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3904.371518] Lustre: 127767:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4201.752938] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:35:06 (1781242506) [ 4217.315413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4217.324062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4217.335843] Lustre: Skipped 5 previous similar messages [ 4217.339803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4217.344131] Lustre: Skipped 8 previous similar messages [ 4221.929154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.953377] Lustre: Skipped 1 previous similar message [ 4223.251492] Lustre: server umount lustre-MDT0000 complete [ 4226.828350] LustreError: 118961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242533 with bad export cookie 10440636093414618705 [ 4226.840638] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.846563] LustreError: 118961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4227.045449] LustreError: 122161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.052349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4227.064661] LustreError: 122161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4227.077043] Lustre: Skipped 5 previous similar messages [ 4241.376886] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4241.787534] Lustre: server umount lustre-MDT0001 complete [ 4260.320149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4260.422474] Lustre: server umount lustre-OST0000 complete [ 4278.243184] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4278.375508] Lustre: server umount lustre-OST0001 complete [ 4285.596095] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4295.044534] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4335.932375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.283705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4336.342485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4336.458744] Lustre: lustre-MDT0000: new disk, initializing [ 4336.545228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4336.552636] Lustre: Skipped 4 previous similar messages [ 4336.573892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4340.617456] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4350.777626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.966617] Lustre: 130795:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4351.079175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4351.087129] Lustre: Skipped 1 previous similar message [ 4351.204628] Lustre: lustre-MDT0001: new disk, initializing [ 4351.385280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4351.396936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4356.215398] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4365.255871] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4369.234895] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4375.552179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.817511] Lustre: lustre-OST0000: new disk, initializing [ 4375.821301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4375.828789] Lustre: 132535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4377.762829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4377.771617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4377.905449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4382.374163] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4391.014552] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4402.004365] Lustre: lustre-MDD0000: changelog on [ 4404.871497] Lustre: lustre-MDD0001: changelog on [ 6338.486366] Lustre: lustre-MDD0001: changelog off [ 6340.466803] Lustre: 134407:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6341.581516] Lustre: lustre-MDD0000: changelog off [ 6349.293974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.294262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6349.295487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6349.295493] Lustre: Skipped 6 previous similar messages [ 6349.301197] Lustre: Skipped 4 previous similar messages [ 6349.328631] LustreError: Skipped 1 previous similar message [ 6353.794620] Lustre: server umount lustre-OST0000 complete [ 6359.525636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.528785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6359.535356] Lustre: Skipped 1 previous similar message [ 6359.540420] Lustre: Skipped 1 previous similar message [ 6364.642762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6364.648780] Lustre: Skipped 1 previous similar message [ 6365.127136] Lustre: server umount lustre-MDT0000 complete [ 6369.018180] LustreError: 130787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244675 with bad export cookie 10440636093415396888 [ 6369.022026] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.026254] LustreError: 130787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6369.536568] Lustre: server umount lustre-MDT0001 complete [ 6381.957723] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6385.409393] Key type lgssc unregistered [ 6385.706628] LNet: 141560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6385.713520] LNetError: 141560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6385.730573] LNet: Removed LNI 192.168.203.123@tcp [ 6386.756157] Key type .llcrypt unregistered [ 6386.758418] Key type ._llcrypt unregistered [ 6410.002904] Key type ._llcrypt registered [ 6410.004807] Key type .llcrypt registered [ 6410.114748] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6427.479278] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6429.248584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6429.304170] alg: No test for adler32 (adler32-zlib) [ 6430.590379] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6431.071189] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6432.768191] Key type lgssc registered [ 6433.867707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6475.175697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6475.235971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.548741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6476.603546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6476.678533] Lustre: lustre-MDT0000: new disk, initializing [ 6476.780919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.804574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6481.743967] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6492.850449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.133244] Lustre: 144731:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6493.195374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6493.200671] Lustre: Skipped 1 previous similar message [ 6493.327716] Lustre: lustre-MDT0001: new disk, initializing [ 6493.493133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6493.606322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6493.624532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6498.196744] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6506.765768] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6511.350845] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6518.363441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6518.653841] Lustre: lustre-OST0000: new disk, initializing [ 6518.656793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6518.664359] Lustre: 146471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6518.742836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.329470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6526.543046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6526.553064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6526.711051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6531.542454] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6538.814930] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_module obdecho/obdecho [ 6540.843204] LustreError: 147349:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6540.856634] LustreError: 147349:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6542.908265] LustreError: 147446:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6555.105450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6555.112794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.127881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.669276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.673919] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.684122] Lustre: Skipped 1 previous similar message [ 6561.058983] Lustre: server umount lustre-MDT0000 complete [ 6564.772489] LustreError: 144721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244871 with bad export cookie 7958886188028420853 [ 6564.773861] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6564.789060] LustreError: 144721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6565.084524] Lustre: server umount lustre-MDT0001 complete [ 6579.658812] Lustre: server umount lustre-OST0000 complete [ 6600.829868] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6610.868853] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6646.970748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.375194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6647.410515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6647.542984] Lustre: lustre-MDT0000: new disk, initializing [ 6647.694821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.732383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6652.413455] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6663.470126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.619907] Lustre: 151125:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6663.689477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6663.701479] Lustre: Skipped 1 previous similar message [ 6663.818924] Lustre: lustre-MDT0001: new disk, initializing [ 6663.946000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.988030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6664.003881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6668.374666] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6676.102154] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6680.471872] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6687.364242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6687.545236] Lustre: lustre-OST0000: new disk, initializing [ 6687.554160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6687.560891] Lustre: 152868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6687.663590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.501838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6694.516069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6694.626349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6695.483304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6704.663590] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6709.262304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6709.549212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6713.477213] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6713.719778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6715.361089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6715.367285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.368777] LustreError: Skipped 1 previous similar message [ 6715.369672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6715.369676] Lustre: Skipped 2 previous similar messages [ 6715.388262] Lustre: Skipped 1 previous similar message [ 6718.436828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.450045] Lustre: Skipped 2 previous similar messages [ 6721.496258] Lustre: server umount lustre-OST0000 complete [ 6725.601738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6725.608647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.624442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6731.185968] Lustre: server umount lustre-MDT0000 complete [ 6733.796782] LustreError: 151133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6733.811803] LustreError: 151133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6734.444478] LustreError: 151118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245041 with bad export cookie 7958886188028421812 [ 6734.448670] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.454953] LustreError: 151118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6734.825780] Lustre: server umount lustre-MDT0001 complete [ 6738.190426] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:17:23 (1781245043) [ 6739.939483] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6741.639861] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:17:26 (1781245046) [ 6749.264792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6749.613743] LustreError: 155133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.672307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.061400] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6754.789176] LustreError: 155132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6759.912045] LustreError: 155133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.361479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.518159] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6773.799676] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6777.857891] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6784.137642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.441712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.454425] Lustre: Skipped 1 previous similar message [ 6789.982416] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6796.121516] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6803.949229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6803.966381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.983884] Lustre: Skipped 1 previous similar message [ 6803.992920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6804.001208] Lustre: Skipped 2 previous similar messages [ 6809.466936] Lustre: server umount lustre-MDT0000 complete [ 6810.083582] LustreError: 155133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.102975] LustreError: 155133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6813.167729] LustreError: 155113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245120 with bad export cookie 7958886188028422694 [ 6813.173621] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6813.179682] LustreError: 155113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6815.201946] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.227819] Lustre: Skipped 2 previous similar messages [ 6818.211272] LustreError: 156244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.235814] LustreError: 156244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6819.598965] Lustre: server umount lustre-MDT0001 complete [ 6825.862851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6839.930504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.216671] LustreError: 158636:0:(llog.c:1646:llog_backup()) MGC192.168.203.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6840.231486] Lustre: 158636:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6843.874464] LustreError: 158641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.690377] LustreError: 158641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.728878] LustreError: 158641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6901.216703] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6901.232542] Lustre: 158636:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6942.208835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6944.363734] LustreError: 158662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6944.391894] LustreError: 158662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 6946.464664] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6947.306627] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6954.199513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.561184] LustreError: 142951:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000308f99fc x1867770481471104/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6958.704097] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (not set up) [ 6958.717158] Lustre: Skipped 7 previous similar messages [ 6962.695199] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6964.208113] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6968.257087] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6969.679451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6973.518471] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6975.082873] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6996.894274] Lustre: server umount lustre-MDT0000 complete [ 7000.036877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7000.043417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.091193] Lustre: Skipped 1 previous similar message [ 7002.563793] LustreError: 155113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245309 with bad export cookie 7958886188028429148 [ 7002.564491] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7002.579051] LustreError: 155113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7003.124471] Lustre: server umount lustre-MDT0001 complete [ 7020.375590] Lustre: server umount lustre-OST0000 complete [ 7043.147199] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7052.951885] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7092.548073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.842339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7092.879573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7092.999803] Lustre: lustre-MDT0000: new disk, initializing [ 7093.146869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.154291] Lustre: Skipped 1 previous similar message [ 7093.190077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7097.552668] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7108.107803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.227448] Lustre: 163479:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7108.446711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7112.940388] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7120.538626] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7124.388565] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7131.161449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.373431] Lustre: lustre-OST0000: new disk, initializing [ 7131.375413] Lustre: Skipped 1 previous similar message [ 7131.378098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7131.384501] Lustre: Skipped 2 previous similar messages [ 7131.399981] Lustre: 165222:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7137.433941] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7140.909942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7140.918652] Lustre: Skipped 1 previous similar message [ 7140.923948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7141.022259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7144.481558] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7148.373634] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7148.530307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7152.052292] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7152.263631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7156.196794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7156.218476] Lustre: Skipped 2 previous similar messages [ 7156.220573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7156.233596] Lustre: Skipped 1 previous similar message [ 7160.258887] Lustre: server umount lustre-OST0000 complete [ 7164.898476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7169.508182] Lustre: server umount lustre-MDT0000 complete [ 7171.554111] LustreError: 163486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7171.581407] LustreError: 163486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 7173.216813] LustreError: 163470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245480 with bad export cookie 7958886188028445983 [ 7173.219170] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7173.226869] LustreError: 163470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7173.514883] Lustre: server umount lustre-MDT0001 complete [ 7187.669664] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:24:50 (1781245490) [ 7199.066620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.749537] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7213.934766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.428461] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7224.238276] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7228.495348] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7234.878391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.202723] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7248.935744] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7261.661919] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:26:06 (1781245566) [ 7269.345272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7269.352436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.366242] Lustre: Skipped 2 previous similar messages [ 7269.371312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7269.377383] Lustre: Skipped 3 previous similar messages [ 7272.832349] Lustre: server umount lustre-OST0000 complete [ 7290.336139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7290.541602] Lustre: server umount lustre-MDT0000 complete [ 7294.032631] LustreError: 169108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245600 with bad export cookie 7958886188028446865 [ 7294.034584] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7294.042393] LustreError: 169108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7294.249680] Lustre: server umount lustre-MDT0001 complete [ 7302.916429] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7305.046306] Key type lgssc unregistered [ 7305.350360] LNet: 170860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7305.359885] LNetError: 170860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7305.383169] LNet: Removed LNI 192.168.203.123@tcp [ 7306.158493] Key type .llcrypt unregistered [ 7306.160572] Key type ._llcrypt unregistered [ 7320.654639] Key type ._llcrypt registered [ 7320.656163] Key type .llcrypt registered [ 7320.754369] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7321.318765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7321.441218] alg: No test for adler32 (adler32-zlib) [ 7322.454955] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7322.657881] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7324.296155] Key type lgssc registered [ 7325.138143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7331.115294] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7331.142479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7346.659339] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7351.777888] LustreError: 172065:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 7356.897068] LustreError: 172065:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7356.926267] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7356.939772] LustreError: MGC192.168.203.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7356.954490] LustreError: 172065:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7356.971661] LustreError: 172065:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7356.980907] LustreError: 172065:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7357.050915] Lustre: server umount lustre-OST0000 complete [ 7357.059417] LustreError: 172065:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7364.619171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.087292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.812543] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7375.354917] LustreError: 172564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.379431] LustreError: 172564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7377.375733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.650204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7380.817824] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7385.337677] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7388.239769] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7393.755724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.929860] LustreError: 174156:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7393.942288] Lustre: 174156:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7394.054499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.513657] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7404.450650] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7407.585768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.594841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7407.600941] Lustre: Skipped 1 previous similar message [ 7407.620544] Lustre: Skipped 2 previous similar messages [ 7412.560152] LustreError: 174802:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000e5dddfe6) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7412.634481] Lustre: server umount lustre-MDT0000 complete [ 7412.708169] LustreError: 172565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7412.723518] LustreError: 172565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7415.553790] LustreError: 172547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245722 with bad export cookie 14394104682554896849 [ 7415.559654] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7415.565399] LustreError: 172547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7415.822719] Lustre: server umount lustre-MDT0001 complete [ 7429.671531] Lustre: server umount lustre-OST0000 complete [ 7446.652104] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7459.865872] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7504.164543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7504.523724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7504.581920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7504.676729] Lustre: lustre-MDT0000: new disk, initializing [ 7504.779698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.809022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7509.719076] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7520.811791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.972368] Lustre: 177537:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7521.035779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7521.038921] Lustre: Skipped 1 previous similar message [ 7521.130481] Lustre: lustre-MDT0001: new disk, initializing [ 7521.246840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7521.293851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7521.304990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7526.552470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7535.979317] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7541.152807] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7548.043243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.270403] Lustre: lustre-OST0000: new disk, initializing [ 7548.273688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7548.279403] Lustre: 179279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7548.370099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7555.573510] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7555.698116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7555.713399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7555.872480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7563.532711] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7568.137227] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7568.418183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7573.181608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7573.402478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7576.553591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7576.559423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7576.577820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7576.581382] Lustre: Skipped 2 previous similar messages [ 7576.589236] LustreError: Skipped 1 previous similar message [ 7581.661932] Lustre: server umount lustre-OST0000 complete [ 7586.794884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.801153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7586.815635] Lustre: Skipped 1 previous similar message [ 7586.843433] Lustre: Skipped 2 previous similar messages [ 7591.905803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7591.916300] Lustre: Skipped 1 previous similar message [ 7597.026314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7597.030028] Lustre: Skipped 1 previous similar message [ 7600.096338] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7600.493776] Lustre: server umount lustre-MDT0000 complete [ 7602.159468] LustreError: 177549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.170503] LustreError: 177549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7604.473962] LustreError: 177528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245911 with bad export cookie 14394104682554897563 [ 7604.477466] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7604.489313] LustreError: 177528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7605.049531] Lustre: server umount lustre-MDT0001 complete [ 7614.548357] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:31:59 (1781245919) [ 7631.482381] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7634.657991] Key type lgssc unregistered [ 7634.999762] LNet: 182061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7635.009132] LNetError: 182061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7635.035935] LNet: Removed LNI 192.168.203.123@tcp [ 7635.862981] Key type .llcrypt unregistered [ 7635.865822] Key type ._llcrypt unregistered [ 7656.382041] Key type ._llcrypt registered [ 7656.385016] Key type .llcrypt registered [ 7656.536780] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7657.873452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7658.005580] alg: No test for adler32 (adler32-zlib) [ 7659.096613] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7659.327428] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7661.056287] Key type lgssc registered [ 7662.665328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7671.826698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7671.854472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.253276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.701849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7683.558867] LustreError: 183262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.590962] LustreError: 183262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7687.495383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.784264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7691.761535] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7697.821230] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7702.651596] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7711.277159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7711.551326] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7711.554569] LustreError: 184902:0:(llog.c:1687:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7711.561351] Lustre: 184902:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7711.727614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7717.994316] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7723.850585] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7728.609994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7728.625359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.647186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.244185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.258279] Lustre: Skipped 5 previous similar messages [ 7739.361500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7739.371906] Lustre: Skipped 2 previous similar messages [ 7740.386364] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7740.737201] Lustre: server umount lustre-MDT0000 complete [ 7744.378465] LustreError: 183981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246051 with bad export cookie 14465345452087193342 [ 7744.380183] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7744.395916] LustreError: 183981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7744.489560] LustreError: 184295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.497534] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.524249] Lustre: Skipped 2 previous similar messages [ 7744.528117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7744.798785] Lustre: server umount lustre-MDT0001 complete [ 7751.160955] Lustre: server umount lustre-OST0000 complete [ 7762.340514] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7775.784579] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7827.508830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7827.727109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7827.747241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7827.806283] Lustre: lustre-MDT0000: new disk, initializing [ 7827.907131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.924940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7832.053368] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7842.629861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.798400] Lustre: 188285:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7842.836348] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7842.841412] Lustre: Skipped 1 previous similar message [ 7842.844077] LustreError: 189049:0:(llog.c:1687:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7842.854180] LustreError: 189049:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7842.861448] Lustre: 189049:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7842.880663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7842.889253] Lustre: Skipped 1 previous similar message [ 7842.948239] Lustre: lustre-MDT0001: new disk, initializing [ 7843.053374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7843.084625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7843.098035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7847.273952] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7856.723159] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7861.389788] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7869.831936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7870.138630] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7870.146021] Lustre: Skipped 1 previous similar message [ 7870.159988] LustreError: 190002:0:(llog.c:1687:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7870.183948] LustreError: 190002:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7870.195784] Lustre: 190002:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7870.203360] Lustre: 190002:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7870.224204] Lustre: lustre-OST0000: new disk, initializing [ 7870.227219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7870.232610] Lustre: 190025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7870.338275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7876.729614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7876.753732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7876.893808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7877.608530] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7887.689815] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7893.329750] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7893.600871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7900.228828] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7900.506728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7902.692393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.702050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.706239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7902.715251] LustreError: Skipped 1 previous similar message [ 7902.730962] Lustre: Skipped 1 previous similar message [ 7908.887895] Lustre: server umount lustre-OST0000 complete [ 7914.978573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7914.984667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7915.002736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7915.015019] Lustre: Skipped 3 previous similar messages [ 7919.749631] Lustre: server umount lustre-MDT0000 complete [ 7923.185866] LustreError: 188291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.215553] LustreError: 188291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7924.433677] LustreError: 189061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246231 with bad export cookie 14465345452087194070 [ 7924.449759] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7924.458143] LustreError: 189061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7925.010784] Lustre: server umount lustre-MDT0001 complete [ 7935.011468] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:37:19 (1781246239) [ 7952.937983] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7962.869020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.161442] LustreError: 192989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.231155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.816628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7968.225980] LustreError: 192990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.392436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.702919] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7983.882396] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7990.235238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.253880] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8007.020276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.286913] Lustre: lustre-OST0001: new disk, initializing [ 8007.298715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8007.306591] Lustre: 195249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8007.445205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8007.456283] Lustre: Skipped 2 previous similar messages [ 8012.957108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8012.967353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8013.199096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8015.918756] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8028.408313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8036.276362] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8038.569863] Lustre: DEBUG MARKER: ADD OST3 [ 8043.482225] loop: module loaded [ 8043.543906] loop0: detected capacity change from 0 to 204800000 [ 8043.731855] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8043.741506] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8043.750922] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8049.722635] loop0: detected capacity change from 0 to 204800000 [ 8050.385593] Lustre: DEBUG MARKER: STOP OST3 [ 8055.838530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8056.094250] Lustre: lustre-OST0003: new disk, initializing [ 8056.098918] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8056.106172] Lustre: 197252:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8057.370113] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8057.540702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8057.548912] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8057.595646] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8059.424535] LustreError: 197361:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8060.297168] Lustre: Failing over lustre-OST0003 [ 8060.505350] Lustre: server umount lustre-OST0003 complete [ 8062.969316] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8062.989613] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.010946] Lustre: Skipped 2 previous similar messages [ 8063.014547] LustreError: 194443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8063.056521] LustreError: 194443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8065.190652] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8078.662542] Lustre: DEBUG MARKER: START OST3 again [ 8083.457966] LustreError: 194441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.503548] LustreError: 194441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8089.402096] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.637282] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8089.650148] Lustre: Skipped 1 previous similar message [ 8089.665699] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8090.869062] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8091.092937] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8091.094104] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8097.301487] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8119.150872] Lustre: Failing over lustre-OST0003 [ 8119.396395] Lustre: server umount lustre-OST0003 complete [ 8120.302907] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.324313] LustreError: 197362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.347319] LustreError: 197362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8140.771406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8140.778631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8140.787858] Lustre: Skipped 4 previous similar messages [ 8140.794648] Lustre: Skipped 4 previous similar messages [ 8144.349558] Lustre: server umount lustre-MDT0000 complete [ 8148.136435] LustreError: 193708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246454 with bad export cookie 14465345452087194952 [ 8148.153624] LustreError: 193708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8148.154676] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8154.746820] Lustre: server umount lustre-MDT0001 complete [ 8158.952931] Lustre: server umount lustre-OST0000 complete [ 8162.709483] Lustre: server umount lustre-OST0001 complete [ 8169.921225] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8179.151375] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8220.404640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.663756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8220.705760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8220.854876] Lustre: lustre-MDT0000: new disk, initializing [ 8220.942070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.983414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8225.149506] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8235.382958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.493761] Lustre: 202632:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8235.836366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8240.423623] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8249.605406] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8254.937887] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8265.975383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8266.577977] Lustre: lustre-OST0000: new disk, initializing [ 8266.587774] Lustre: Skipped 1 previous similar message [ 8266.595866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8266.603016] Lustre: Skipped 2 previous similar messages [ 8266.619961] Lustre: 204374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8268.422584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8268.449922] Lustre: Skipped 1 previous similar message [ 8268.455620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8268.587332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8277.089488] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8289.357980] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8295.303163] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8295.530302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8300.489995] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8300.641565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8304.616039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8304.619535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8304.624575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8304.624887] Lustre: Skipped 3 previous similar messages [ 8304.630416] Lustre: Skipped 6 previous similar messages [ 8308.957282] Lustre: server umount lustre-OST0000 complete [ 8322.980638] Lustre: server umount lustre-MDT0000 complete [ 8325.091772] LustreError: 202642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8325.149335] LustreError: 202642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 8328.424772] LustreError: 203408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246635 with bad export cookie 14465345452087197248 [ 8328.427693] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8328.443092] LustreError: 203408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8329.180824] Lustre: server umount lustre-MDT0001 complete [ 8334.857693] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:43:59 (1781246639) [ 8353.947744] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8364.948535] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8403.199797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.502159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8403.553252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8403.647377] Lustre: lustre-MDT0000: new disk, initializing [ 8403.793479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8408.217892] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8419.078666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8419.219329] Lustre: 208806:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8419.509340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8424.287846] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8432.771140] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8437.408580] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8445.109708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8445.488792] Lustre: 210549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8447.693600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8452.199331] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8461.497318] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8466.620626] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8466.894495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8472.214466] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8472.411597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8478.689551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8478.691856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8478.701961] LustreError: Skipped 2 previous similar messages [ 8478.717215] Lustre: Skipped 2 previous similar messages [ 8478.718311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8478.718315] Lustre: Skipped 4 previous similar messages [ 8480.491678] Lustre: server umount lustre-OST0000 complete [ 8491.901483] Lustre: server umount lustre-MDT0000 complete [ 8494.050500] LustreError: 210645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.094654] LustreError: 210645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8496.810869] LustreError: 209582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246803 with bad export cookie 14465345452087198130 [ 8496.820368] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.826816] LustreError: 209582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8497.176742] Lustre: server umount lustre-MDT0001 complete [ 8507.608037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.261826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8508.269656] Lustre: Skipped 5 previous similar messages [ 8514.562593] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8526.226555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.823569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8539.215852] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8543.507651] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8551.967643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8560.370335] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8568.834787] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8582.130809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.137581] LustreError: Skipped 1 previous similar message [ 8594.400228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8594.604661] Lustre: server umount lustre-OST0000 complete [ 8604.631256] Lustre: server umount lustre-MDT0000 complete [ 8608.610575] LustreError: 212601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246915 with bad export cookie 14465345452087199012 [ 8608.625263] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8608.632540] LustreError: 212601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8609.093747] Lustre: server umount lustre-MDT0001 complete [ 8709.833977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8713.455467] Key type lgssc unregistered [ 8713.800854] LNet: 215928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8713.825563] LNetError: 215928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8714.855641] LNet: Removed LNI 192.168.203.123@tcp [ 8715.827285] Key type .llcrypt unregistered [ 8715.832751] Key type ._llcrypt unregistered [ 8742.688238] Key type ._llcrypt registered [ 8742.692610] Key type .llcrypt registered [ 8742.798264] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8768.623540] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8770.434567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8770.687169] alg: No test for adler32 (adler32-zlib) [ 8772.066478] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 8772.312480] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8773.960300] Key type lgssc registered [ 8775.415934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8815.942453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8815.967234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.146548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8817.166081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8817.260825] Lustre: lustre-MDT0000: new disk, initializing [ 8817.374993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.402369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8821.236438] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8830.537795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.663205] Lustre: 219249:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8830.749571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8830.763326] Lustre: Skipped 1 previous similar message [ 8830.835512] Lustre: lustre-MDT0001: new disk, initializing [ 8831.053558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8831.128235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8831.147031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8835.708348] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8842.797377] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8846.160948] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8853.351500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8853.656605] Lustre: lustre-OST0000: new disk, initializing [ 8853.662383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8853.678429] Lustre: 220991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8853.820627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8860.655836] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8862.278170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8862.287727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8862.389620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8869.638141] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8874.119226] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8874.331554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8878.393572] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8878.651728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8883.169900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8883.171772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.184883] LustreError: Skipped 1 previous similar message [ 8883.197305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8883.220614] Lustre: Skipped 1 previous similar message [ 8886.550382] Lustre: server umount lustre-OST0000 complete [ 8892.385372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8892.400453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8892.408230] Lustre: Skipped 1 previous similar message [ 8892.412459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8893.411502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.423065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8893.427879] Lustre: Skipped 1 previous similar message [ 8897.970482] Lustre: server umount lustre-MDT0000 complete [ 8902.172175] LustreError: 220038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247209 with bad export cookie 3701703129675904544 [ 8902.182852] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8902.188833] LustreError: 220038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8902.562602] Lustre: server umount lustre-MDT0001 complete [ 8922.962409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8935.113138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8935.812966] LustreError: 223762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.833559] LustreError: 223762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8935.864510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.041618] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8941.036735] LustreError: 223763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.149691] LustreError: 223762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.373839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.664415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8952.663135] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8955.707378] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8962.468909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8962.870866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8970.709535] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8982.043379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8982.216975] Lustre: lustre-OST0001: new disk, initializing [ 8982.226755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8982.239422] Lustre: 226021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8982.368807] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.604776] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8991.454058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8991.470195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8991.667500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9002.467982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9010.612615] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9024.297241] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:55:29 (1781247329) [ 9027.558087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.568435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.583812] Lustre: Skipped 2 previous similar messages [ 9027.596782] Lustre: Skipped 3 previous similar messages [ 9032.687248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9032.696342] Lustre: Skipped 3 previous similar messages [ 9040.352699] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9040.561412] Lustre: server umount lustre-MDT0000 complete [ 9042.927754] LustreError: 223763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.941512] LustreError: 223763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9043.819966] LustreError: 224864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247350 with bad export cookie 3701703129675905419 [ 9043.821750] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9043.827848] LustreError: 224864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9044.295873] Lustre: server umount lustre-MDT0001 complete [ 9057.384493] Lustre: server umount lustre-OST0000 complete [ 9071.030233] Lustre: server umount lustre-OST0001 complete [ 9076.029446] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9084.323326] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9116.747203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.962796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9116.983713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9117.046555] Lustre: lustre-MDT0000: new disk, initializing [ 9117.131786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9117.149342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9120.824583] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9130.252232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9130.365753] Lustre: 229924:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9130.437801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9130.441581] Lustre: Skipped 1 previous similar message [ 9130.508165] Lustre: lustre-MDT0001: new disk, initializing [ 9130.614766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9130.635454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9134.711624] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9141.946443] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9145.444290] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9151.366810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9151.640841] Lustre: lustre-OST0000: new disk, initializing [ 9151.644627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9151.649846] Lustre: 231663:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9153.647826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9153.670625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9153.756284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9156.182929] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9162.697370] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9166.306573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9166.520995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9170.367297] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9170.559484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9174.501744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9174.505605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.513996] LustreError: Skipped 1 previous similar message [ 9174.522507] Lustre: Skipped 1 previous similar message [ 9174.531684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9174.541158] Lustre: Skipped 4 previous similar messages [ 9178.340798] Lustre: server umount lustre-OST0000 complete [ 9181.665396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9184.740116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9184.753506] Lustre: Skipped 2 previous similar messages [ 9187.388425] Lustre: server umount lustre-MDT0000 complete [ 9189.860180] LustreError: 229934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.870446] LustreError: 229934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9190.622404] LustreError: 229916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247497 with bad export cookie 3701703129675907561 [ 9190.623812] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9190.631071] LustreError: 229916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9190.944411] Lustre: server umount lustre-MDT0001 complete [ 9197.334969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.671989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9197.677858] Lustre: Skipped 2 previous similar messages [ 9200.781565] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9203.172947] LustreError: 233730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.183642] LustreError: 233730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9207.587234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9210.632503] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9214.702290] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9217.472606] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9221.933117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9225.744411] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9229.157469] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9375.715937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9375.717399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.720181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9375.720188] Lustre: Skipped 3 previous similar messages [ 9375.735817] Lustre: Skipped 1 previous similar message [ 9379.755990] Lustre: server umount lustre-OST0000 complete [ 9396.192130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9396.291736] Lustre: server umount lustre-MDT0000 complete [ 9398.124517] LustreError: 233711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247704 with bad export cookie 3701703129675908443 [ 9398.126941] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9398.131078] LustreError: 233711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9398.284775] Lustre: server umount lustre-MDT0001 complete [ 9495.040445] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9496.531955] Key type lgssc unregistered [ 9496.689760] LNet: 236992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9496.694646] LNetError: 236992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9496.705549] LNet: Removed LNI 192.168.203.123@tcp [ 9497.105157] Key type .llcrypt unregistered [ 9497.107508] Key type ._llcrypt unregistered [ 9509.885159] Key type ._llcrypt registered [ 9509.886942] Key type .llcrypt registered [ 9509.943606] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9518.628966] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9519.227396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9519.240816] alg: No test for adler32 (adler32-zlib) [ 9520.152551] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 9520.288158] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9521.912182] Key type lgssc registered [ 9522.503088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9545.561273] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9550.265391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9550.277201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.409991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9551.428878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9551.470819] Lustre: lustre-MDT0000: new disk, initializing [ 9551.508479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.519182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9553.262807] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9558.540634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.592986] Lustre: 240982:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9558.624703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9558.627359] Lustre: Skipped 1 previous similar message [ 9558.671347] Lustre: lustre-MDT0001: new disk, initializing [ 9558.714397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9558.736595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9558.742213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9560.517332] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9563.244476] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9566.321119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9566.490630] Lustre: lustre-OST0000: new disk, initializing [ 9566.494241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9566.498129] Lustre: 242578:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9566.548540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9567.833182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9567.837783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9567.907558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9569.281802] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9574.598875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9574.687264] Lustre: lustre-OST0001: new disk, initializing [ 9574.689836] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9574.694908] Lustre: 243432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9574.749710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9577.330492] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9577.815127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9577.821269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9577.884702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9584.687217] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9587.879254] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9593.315757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.320768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9593.326907] Lustre: Skipped 3 previous similar messages [ 9593.331756] Lustre: Skipped 3 previous similar messages [ 9598.470906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9598.480510] Lustre: Skipped 3 previous similar messages [ 9599.817309] Lustre: server umount lustre-MDT0000 complete [ 9604.931708] LustreError: 243442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247911 with bad export cookie 4197382763160554280 [ 9604.959088] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9604.964123] LustreError: 243442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9605.562886] Lustre: server umount lustre-MDT0001 complete [ 9621.126869] Lustre: server umount lustre-OST0000 complete [ 9625.061864] Lustre: 238509:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781247915/real 1781247915] req@ffff8a42d43b0f00 x1867773721510784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781247931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9625.115584] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9626.202863] Lustre: server umount lustre-OST0001 complete [ 9647.021266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9664.845220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9678.555461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9688.053584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9705.932849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9715.850262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9715.872492] Lustre: lustre-MDT0000: reset Object Index mappings [ 9716.172931] LustreError: 248408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9716.184226] LustreError: 248408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9716.209814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.503489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9721.315123] LustreError: 248409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.437770] LustreError: 248408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9728.076558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.092959] Lustre: lustre-MDT0001: reset Object Index mappings [ 9728.465381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9731.928969] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9734.594192] Lustre: 249519:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9740.779508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.135849] LustreError: 249872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.149137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9747.274295] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9747.435480] LustreError: 249872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9755.407204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9755.675282] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9755.684318] Lustre: Skipped 1 previous similar message [ 9760.775353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9761.848748] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9769.430397] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9773.037159] Lustre: 251355:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9779.685438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9779.694694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.709483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.947158] Lustre: server umount lustre-MDT0000 complete [ 9781.227473] LustreError: 249126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9781.243858] LustreError: 249126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9783.651493] LustreError: 248390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248090 with bad export cookie 4197382763160561056 [ 9783.656753] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9783.665348] LustreError: 248390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9784.029739] Lustre: server umount lustre-MDT0001 complete [ 9797.850634] Lustre: server umount lustre-OST0000 complete [ 9811.815802] Lustre: server umount lustre-OST0001 complete [ 9829.633867] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:08:54 (1781248134) [ 9845.180416] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9854.545952] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9889.097131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.344243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9889.387514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9889.467741] Lustre: lustre-MDT0000: new disk, initializing [ 9889.532563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9889.550526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9892.955434] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9902.776967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.882865] Lustre: 255574:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9902.929883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9902.933579] Lustre: Skipped 1 previous similar message [ 9902.998337] Lustre: lustre-MDT0001: new disk, initializing [ 9903.109119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9903.119185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9907.307724] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9914.778144] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9918.909821] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9925.003273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9925.286559] Lustre: lustre-OST0000: new disk, initializing [ 9925.295251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9925.304602] Lustre: 257316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9926.937539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9926.950041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9927.063189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9930.302847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9936.036823] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9939.506872] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9939.685634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9943.400592] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9943.575946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9947.623283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9947.632520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9947.636381] LustreError: Skipped 1 previous similar message [ 9947.637652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9947.663201] Lustre: Skipped 4 previous similar messages [ 9952.745615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9952.754235] Lustre: Skipped 2 previous similar messages [ 9959.392349] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9959.868538] Lustre: server umount lustre-OST0000 complete [ 9969.634658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9969.645533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.669071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9969.681038] Lustre: Skipped 2 previous similar messages [ 9975.027526] Lustre: server umount lustre-MDT0000 complete [ 9978.341071] LustreError: 255586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9978.387479] LustreError: 255586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9981.812846] LustreError: 255565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248288 with bad export cookie 4197382763160563149 [ 9981.818117] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9981.841793] LustreError: 255565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9982.458245] Lustre: server umount lustre-MDT0001 complete [10004.111764] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10014.020073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.320111] LustreError: 260088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.359293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10014.367724] Lustre: Skipped 2 previous similar messages [10019.436980] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10027.891393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10032.617051] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10035.500396] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10042.551508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10049.436678] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10058.232256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10058.480122] Lustre: lustre-OST0001: new disk, initializing [10058.483875] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10058.499987] Lustre: 262348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10060.296680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10060.313186] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10060.445297] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10064.704161] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10074.168193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10077.584841] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10083.355858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10083.359560] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10091.491186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10091.492457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10091.508314] Lustre: Skipped 3 previous similar messages [10091.515423] Lustre: Skipped 4 previous similar messages [10097.110806] Lustre: server umount lustre-MDT0000 complete [10101.049551] LustreError: 260068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248407 with bad export cookie 4197382763160564031 [10101.060571] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10101.576925] Lustre: server umount lustre-MDT0001 complete [10115.805071] Lustre: server umount lustre-OST0000 complete [10118.113102] Lustre: 238509:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781248408/real 1781248408] req@ffff8a43c5b2fc00 x1867773721666688/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781248424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10118.152788] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10118.167838] Lustre: Skipped 1 previous similar message [10119.722848] Lustre: server umount lustre-OST0001 complete [10141.474235] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10152.932590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.185335] LustreError: 265370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.201967] LustreError: 265370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10153.261060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10153.265449] Lustre: Skipped 3 previous similar messages [10158.216069] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10167.401954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10172.303625] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10175.725464] Lustre: 266476:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10182.119568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10188.372305] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10197.394111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.727870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10198.756344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10204.314845] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10211.995364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10215.983755] Lustre: 268318:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10221.712312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10221.806154] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10232.910706] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:15:37 (1781248537) [10249.698114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10249.726436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10249.806713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10249.814207] Lustre: Skipped 5 previous similar messages [10251.507921] Lustre: server umount lustre-MDT0000 complete [10257.799941] LustreError: 265350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248564 with bad export cookie 4197382763160570583 [10257.818608] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.826754] LustreError: 265350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10272.225313] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10272.758290] Lustre: server umount lustre-MDT0001 complete [10285.183324] Lustre: server umount lustre-OST0000 complete [10289.932529] Lustre: server umount lustre-OST0001 complete [10297.439372] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [10308.025689] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10366.420677] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10377.877364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10378.206820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10378.230932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10378.325366] Lustre: lustre-MDT0000: new disk, initializing [10378.495462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10384.161699] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10395.697340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10395.817915] Lustre: 272510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10396.108638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10400.989746] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10414.491884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10414.721647] Lustre: 274112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10414.863694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10414.874775] Lustre: Skipped 5 previous similar messages [10416.714026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10421.959073] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10434.133504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.443502] Lustre: 274965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10436.229355] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10436.239607] Lustre: Skipped 1 previous similar message [10436.294595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10441.352489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10451.870238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10456.160435] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10456.166476] Lustre: Skipped 1 previous similar message [10473.294419] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:19:37 (1781248777) [10477.543456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10477.548958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10477.556826] Lustre: Skipped 6 previous similar messages [10477.562466] Lustre: Skipped 10 previous similar messages [10477.736241] Lustre: server umount lustre-MDT0000 complete [10481.195443] LustreError: 272504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248788 with bad export cookie 4197382763160578220 [10481.200033] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10481.212046] LustreError: 272504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10481.594205] Lustre: server umount lustre-MDT0001 complete [10495.564623] Lustre: server umount lustre-OST0000 complete [10498.534928] Lustre: 238512:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781248789/real 1781248789] req@ffff8a42c487cf00 x1867773721818240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781248805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10499.777557] Lustre: server umount lustre-OST0001 complete [10508.991688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10509.509895] LustreError: 277193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.526707] LustreError: 277193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [10514.001671] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10523.198811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.929508] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10533.780677] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10537.572490] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10544.077819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10550.842218] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10556.653993] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10572.099315] loop1: detected capacity change from 0 to 204800000 [10572.304145] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10572.334740] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10572.356271] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10577.159548] loop1: detected capacity change from 0 to 204800000 [10582.596688] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10582.714152] LustreError: 277183:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs15749: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10582.734105] LustreError: 277183:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs15749 [10582.743267] LustreError: 277183:0:(mgs_handler.c:517:mgs_target_reg()) fs15749-MDT0001: Error checking target index: rc = -13 [10582.758297] LustreError: fs15749-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10582.765898] LustreError: 280571:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10582.771311] LustreError: 280571:0:(tgt_mount.c:2084:server_put_super()) no obd fs15749-MDT0001 [10582.778931] LustreError: 280571:0:(tgt_mount.c:116:server_deregister_mount()) fs15749-MDT0001 not registered [10582.810581] Lustre: server umount fs15749-MDT0001 complete [10582.813465] LustreError: 280571:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10590.338159] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10590.419749] Lustre: Setting parameter fs15749-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15749-MDT0001 [10590.464317] Lustre: srv-fs15749-MDT0001: No data found on store. Initialize space. [10590.467951] Lustre: Skipped 4 previous similar messages [10590.537263] Lustre: fs15749-MDT0001: new disk, initializing [10590.543311] Lustre: Skipped 3 previous similar messages [10590.670030] Lustre: srv-fs15749-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10590.679869] Lustre: 281037:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15749-MDT0001-mdtlov: can't create new llog: rc = -115 [10590.688230] LustreError: 281037:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15749-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10594.905490] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10600.523237] Lustre: server umount fs15749-MDT0001 complete [10611.522402] loop1: detected capacity change from 0 to 204800000 [10616.123889] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.518064] Lustre: 282589:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15749-MDT0001-mdtlov: can't create new llog: rc = -115 [10616.532895] LustreError: 282589:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15749-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10620.917736] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10625.396292] Lustre: server umount fs15749-MDT0001 complete [10631.660757] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10634.379627] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [10634.388717] Lustre: Skipped 3 previous similar messages [10636.260635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10636.280479] Lustre: Skipped 3 previous similar messages [10640.467879] Lustre: server umount lustre-MDT0000 complete [10647.762100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10647.851276] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10651.480316] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10653.167170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10657.611094] loop1: detected capacity change from 0 to 204800000 [10662.623198] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.120779] Lustre: 285002:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15749-MDT0001-mdtlov: can't create new llog: rc = -115 [10663.131693] LustreError: 285002:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15749-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10666.681326] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10671.044315] Lustre: server umount fs15749-MDT0001 complete [10692.065493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10692.071691] LustreError: Skipped 1 previous similar message [10703.329083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10703.472504] Lustre: server umount lustre-OST0000 complete [10709.475722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10709.483066] LustreError: Skipped 1 previous similar message [10712.717580] Lustre: server umount lustre-MDT0000 complete [10715.559989] LustreError: 277175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249022 with bad export cookie 4197382763160580901 [10715.564024] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10715.570481] LustreError: 277175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10715.878665] Lustre: server umount lustre-MDT0001 complete [10725.248930] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [10727.974590] Key type lgssc unregistered [10728.186799] LNet: 287311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10728.199788] LNetError: 287311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10728.213729] LNet: Removed LNI 192.168.203.123@tcp [10728.860884] Key type .llcrypt unregistered [10728.881519] Key type ._llcrypt unregistered [10744.433968] Key type ._llcrypt registered [10744.437938] Key type .llcrypt registered [10744.535098] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10745.502587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10745.540228] alg: No test for adler32 (adler32-zlib) [10746.536425] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [10746.708801] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [10748.368352] Key type lgssc registered [10749.156530] Lustre: Echo OBD driver; http://www.lustre.org/ [10755.508536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10755.536627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10756.983870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10760.521219] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10766.313142] LustreError: 288557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.335497] LustreError: 288557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10766.824400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10767.149372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10770.040991] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10774.206852] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10776.951805] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10782.068083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10782.326407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10786.722802] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10791.047265] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10791.531218] LustreError: 290159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.033255] LustreError: 290159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.153052] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.173048] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10807.267362] LustreError: 290160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.281915] LustreError: 290160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10816.480631] loop1: detected capacity change from 0 to 204800000 [10816.615488] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10816.635151] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10816.645213] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10817.507103] LustreError: 290159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.519260] LustreError: 290159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10820.927297] loop1: detected capacity change from 0 to 204800000 [10825.326396] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10825.418455] Lustre: fs2666-OST0000: new disk, initializing [10825.421490] Lustre: srv-fs2666-OST0000: No data found on store. Initialize space. [10825.427915] Lustre: 291973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10825.476232] Lustre: fs2666-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10830.142126] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10833.541475] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10833.557173] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10837.138206] Lustre: server umount fs2666-OST0000 complete [10846.840269] loop1: detected capacity change from 0 to 204800000 [10851.179229] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10851.393608] Lustre: fs2666-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10856.258210] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10862.883119] loop2: detected capacity change from 0 to 204800000 [10863.050858] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10863.063191] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10863.082307] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10866.975995] loop2: detected capacity change from 0 to 204800000 [10868.709564] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10868.728419] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [10871.244535] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10871.322805] LustreError: fs2666-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10871.337917] LustreError: fs2666-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10871.352734] LustreError: 294978:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10871.360349] LustreError: 294978:0:(tgt_mount.c:2084:server_put_super()) no obd fs2666-OST0001 [10871.365400] LustreError: 294978:0:(tgt_mount.c:116:server_deregister_mount()) fs2666-OST0001 not registered [10871.394985] Lustre: server umount fs2666-OST0001 complete [10871.397939] LustreError: 294978:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10878.957132] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10879.339485] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10879.438410] Lustre: fs2666-OST0001: new disk, initializing [10879.452760] Lustre: srv-fs2666-OST0001: No data found on store. Initialize space. [10879.463601] Lustre: 295528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10879.522621] Lustre: fs2666-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10884.073195] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10884.379585] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10891.198887] Lustre: server umount fs2666-OST0000 complete [10898.110585] Lustre: server umount fs2666-OST0001 complete [10907.770500] loop1: detected capacity change from 0 to 204800000 [10912.381659] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10912.554538] Lustre: fs2666-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10916.872508] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10923.243991] loop2: detected capacity change from 0 to 204800000 [10930.497073] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.275449] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.308816] LustreError: 290162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [10940.545644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10953.944615] Lustre: server umount fs2666-OST0000 complete [10966.383575] Lustre: server umount fs2666-OST0001 complete [10986.465680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10986.471705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10986.487568] LustreError: Skipped 1 previous similar message [10986.492190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10986.559534] Lustre: Skipped 1 previous similar message [10988.288467] Lustre: server umount lustre-OST0000 complete [10996.706117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.728367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10996.731729] Lustre: Skipped 1 previous similar message [10999.928272] Lustre: server umount lustre-MDT0000 complete [11005.958551] LustreError: 288538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249312 with bad export cookie 10474601426595758339 [11005.967211] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11005.975727] LustreError: 288538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11006.571245] Lustre: server umount lustre-MDT0001 complete [11018.247679] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11021.470833] Key type lgssc unregistered [11021.788522] LNet: 301383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11021.805475] LNetError: 301383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11021.830272] LNet: Removed LNI 192.168.203.123@tcp [11022.883364] Key type .llcrypt unregistered [11022.886412] Key type ._llcrypt unregistered [11053.891049] Key type ._llcrypt registered [11053.892659] Key type .llcrypt registered [11054.030482] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11068.281519] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11069.180437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11069.294340] alg: No test for adler32 (adler32-zlib) [11070.423236] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11070.633799] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11072.336414] Key type lgssc registered [11073.464892] Lustre: Echo OBD driver; http://www.lustre.org/ [11109.673776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11109.695392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11110.919727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11110.968602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11111.036821] Lustre: lustre-MDT0000: new disk, initializing [11111.102917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11111.122813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11115.629433] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11125.222938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11125.345249] Lustre: 305210:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11125.399968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11125.404238] Lustre: Skipped 1 previous similar message [11125.503341] Lustre: lustre-MDT0001: new disk, initializing [11125.594211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11125.655988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11125.662785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11129.873944] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11137.332832] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11140.763875] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11147.661263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11147.900757] Lustre: lustre-OST0000: new disk, initializing [11147.911138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11147.931259] Lustre: 306951:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11148.044977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11155.131666] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11155.544491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11155.558366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11155.647672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11164.903647] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11167.906733] Lustre: Mounted lustre-client [11169.253439] Lustre: Unmounted lustre-client [11181.538174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11181.541593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11181.544386] LustreError: Skipped 1 previous similar message [11181.545764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11181.581120] Lustre: Skipped 1 previous similar message [11183.589956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11183.593548] Lustre: Skipped 2 previous similar messages [11188.723781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11188.727761] Lustre: Skipped 1 previous similar message [11193.826546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11193.836976] Lustre: Skipped 1 previous similar message [11194.336333] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11194.465057] Lustre: server umount lustre-OST0000 complete [11198.948172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11198.960832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11198.965947] Lustre: Skipped 1 previous similar message [11204.093334] Lustre: server umount lustre-MDT0000 complete [11208.117758] LustreError: 305201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249514 with bad export cookie 4234735228190497860 [11208.130504] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11208.359340] Lustre: server umount lustre-MDT0001 complete [11219.259594] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11222.298767] Key type lgssc unregistered [11226.336164] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp [11226.720481] LNet: 308982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11226.734847] LNetError: 308982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11298.418434] LNet: Removed LNI 192.168.203.123@tcp [11299.322821] Key type .llcrypt unregistered [11299.324283] Key type ._llcrypt unregistered [11317.183071] Key type ._llcrypt registered [11317.186558] Key type .llcrypt registered [11317.288475] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11331.931910] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11332.963699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11333.256939] alg: No test for adler32 (adler32-zlib) [11334.305781] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11334.524977] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11336.176171] Key type lgssc registered [11337.265445] Lustre: Echo OBD driver; http://www.lustre.org/ [11367.266675] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:34:32 (1781249672) [11373.789946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11373.807916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.134928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11375.202708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11375.326537] Lustre: lustre-MDT0000: new disk, initializing [11375.419156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11375.464611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11379.443474] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11389.060845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11389.199334] Lustre: 312175:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11389.270109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11389.273644] Lustre: Skipped 1 previous similar message [11389.361631] Lustre: lustre-MDT0001: new disk, initializing [11389.467754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11389.553954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11389.572430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11393.997782] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11401.412785] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11406.207205] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11413.335549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11413.618903] Lustre: lustre-OST0000: new disk, initializing [11413.622633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11413.631764] Lustre: 313918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11413.731540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11419.594698] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11422.267429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11422.284569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11422.447442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11426.563528] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11438.048996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11438.057164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11438.065639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11438.076164] Lustre: Skipped 1 previous similar message [11442.350695] Lustre: server umount lustre-OST0000 complete [11448.289665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11448.294378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11448.299923] Lustre: Skipped 1 previous similar message [11448.307491] Lustre: Skipped 1 previous similar message [11452.263732] Lustre: server umount lustre-MDT0000 complete [11455.771030] LustreError: 312166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249762 with bad export cookie 13629782034112933701 [11455.775680] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11455.784796] LustreError: 312166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11456.145539] Lustre: server umount lustre-MDT0001 complete [11468.154430] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11470.680485] Key type lgssc unregistered [11470.911862] LNet: 315545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11470.925652] LNetError: 315545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11470.936964] LNet: Removed LNI 192.168.203.123@tcp [11471.794688] Key type .llcrypt unregistered [11471.801273] Key type ._llcrypt unregistered [11501.535260] Key type ._llcrypt registered [11501.537376] Key type .llcrypt registered [11501.636818] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11502.858075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11503.082376] alg: No test for adler32 (adler32-zlib) [11504.182982] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11504.452104] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11506.208783] Key type lgssc registered [11507.282640] Lustre: Echo OBD driver; http://www.lustre.org/ [11515.863337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11515.881757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11517.592336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11522.154299] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11528.163819] LustreError: 316951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11528.184369] LustreError: 316951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11530.634490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11530.945115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11534.850434] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11540.398674] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11544.090987] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11550.566862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11550.920931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11556.723586] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11562.054065] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11573.455575] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11574.789272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11578.505875] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [11578.854164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.871361] Lustre: Skipped 2 previous similar messages [11583.598635] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [11583.607225] Lustre: Skipped 3 previous similar messages [11588.727882] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [11588.736165] Lustre: Skipped 3 previous similar messages [11592.161111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11592.287165] Lustre: server umount lustre-MDT0000 complete [11593.838976] LustreError: 316947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11595.557897] LustreError: 316931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249902 with bad export cookie 6871107024018564022 [11595.560321] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11595.572254] LustreError: 316931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11598.962624] LustreError: 316946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.978954] LustreError: 316946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11598.998993] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [11599.011223] Lustre: Skipped 3 previous similar messages [11599.332282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11601.941924] Lustre: server umount lustre-MDT0001 complete [11605.655721] Lustre: server umount lustre-OST0000 complete [11612.867049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11616.298036] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11618.807914] Lustre: 320679:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.123@tcp [11620.564812] Lustre: server umount MGS complete [11627.784505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11628.230793] LustreError: 321250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.246908] LustreError: 321250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11628.292731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11631.319684] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11633.634314] LustreError: 321251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11638.144169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11638.447669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11641.743930] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11648.679949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11648.975290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11654.093865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11658.899938] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11660.250769] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11663.212422] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11664.768671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11670.498380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11670.504351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11670.507384] LustreError: Skipped 1 previous similar message [11670.534384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11670.537572] Lustre: Skipped 2 previous similar messages [11670.759482] Lustre: server umount lustre-OST0000 complete [11678.695536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11678.701897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.714874] Lustre: Skipped 1 previous similar message [11680.549098] Lustre: server umount lustre-MDT0000 complete [11683.647923] LustreError: 321231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249990 with bad export cookie 6871107024018564869 [11683.655520] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11683.665647] LustreError: 321231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11694.191082] Lustre: server umount lustre-MDT0001 complete [11703.159088] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11706.083828] Key type lgssc unregistered [11706.384972] LNet: 324425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11706.392939] LNetError: 324425:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11707.435389] LNet: Removed LNI 192.168.203.123@tcp [11708.131505] Key type .llcrypt unregistered [11708.134674] Key type ._llcrypt unregistered [11722.184179] Key type ._llcrypt registered [11722.186370] Key type .llcrypt registered [11722.267819] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11734.043848] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11735.023334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11735.038630] alg: No test for adler32 (adler32-zlib) [11736.040548] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11736.234761] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11737.881120] Key type lgssc registered [11738.908763] Lustre: Echo OBD driver; http://www.lustre.org/ [11766.977541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11767.002179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.234173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11768.264929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11768.343934] Lustre: lustre-MDT0000: new disk, initializing [11768.421715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11768.436181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11771.347284] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11779.984194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11780.165817] Lustre: 327531:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11780.232585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11780.235442] Lustre: Skipped 1 previous similar message [11780.290793] Lustre: lustre-MDT0001: new disk, initializing [11780.398691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11780.426552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11780.442357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11783.887973] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11790.165970] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11792.885798] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11797.720964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11797.917285] Lustre: lustre-OST0000: new disk, initializing [11797.922235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11797.927499] Lustre: 329275:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11798.003531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11802.489910] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11806.247128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11806.260500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11806.335977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11808.391708] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11811.693547] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11811.866438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11815.042272] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11815.196572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11816.929922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11816.938469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.963032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11821.539242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11821.543162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.546214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11822.836056] Lustre: server umount lustre-OST0000 complete [11826.662759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11826.663148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11826.683239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11826.690018] Lustre: Skipped 2 previous similar messages [11831.777620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11831.784121] Lustre: Skipped 1 previous similar message [11832.091841] Lustre: server umount lustre-MDT0000 complete [11834.891690] LustreError: 328318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250141 with bad export cookie 4090973155671428470 [11834.897698] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11834.901166] LustreError: 328318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11835.126895] Lustre: server umount lustre-MDT0001 complete [11841.423750] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:42:26 (1781250146) [11852.984458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11860.269885] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11887.410841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11887.633272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11887.689209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11887.773918] Lustre: lustre-MDT0000: new disk, initializing [11887.845306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11887.862572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11890.881609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11900.707939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.791175] Lustre: 333848:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11900.827390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11900.830468] Lustre: Skipped 1 previous similar message [11900.877329] Lustre: lustre-MDT0001: new disk, initializing [11900.948957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11900.966417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11900.972807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11906.338995] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11912.635880] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11915.410144] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11920.240371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11920.452361] Lustre: lustre-OST0000: new disk, initializing [11920.455183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11920.458802] Lustre: 335590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11920.523321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11924.979172] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11928.609826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11928.627560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11928.733552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11931.376134] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11959.265381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11959.279106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.294667] Lustre: Skipped 1 previous similar message [11959.298466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11964.147872] Lustre: server umount lustre-OST0000 complete [11967.459496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11967.465285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11967.479059] Lustre: Skipped 1 previous similar message [11967.486653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11967.493628] Lustre: Skipped 1 previous similar message [11973.533623] Lustre: server umount lustre-MDT0000 complete [11976.959022] LustreError: 334625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250283 with bad export cookie 4090973155671429352 [11976.964272] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11976.976463] LustreError: 334625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11977.331448] Lustre: server umount lustre-MDT0001 complete [11986.308419] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11988.915289] Key type lgssc unregistered [11989.143504] LNet: 337609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11989.148042] LNetError: 337609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11989.156988] LNet: Removed LNI 192.168.203.123@tcp [11989.838249] Key type .llcrypt unregistered [11989.842202] Key type ._llcrypt unregistered [12056.542789] Key type ._llcrypt registered [12056.546733] Key type .llcrypt registered [12056.625600] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12056.838973] Key type .llcrypt unregistered [12056.845430] Key type ._llcrypt unregistered [12116.549742] Key type ._llcrypt registered [12116.551711] Key type .llcrypt registered [12116.654959] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12116.898704] Key type .llcrypt unregistered [12116.901125] Key type ._llcrypt unregistered [12140.955351] Key type ._llcrypt registered [12140.957527] Key type .llcrypt registered [12141.078691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12141.357359] Key type .llcrypt unregistered [12141.360655] Key type ._llcrypt unregistered [12166.067894] Key type ._llcrypt registered [12166.071087] Key type .llcrypt registered [12166.200856] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12166.507299] Key type .llcrypt unregistered [12166.509197] Key type ._llcrypt unregistered [12228.442496] Key type ._llcrypt registered [12228.445431] Key type .llcrypt registered [12228.544224] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12228.792396] Key type .llcrypt unregistered [12228.795521] Key type ._llcrypt unregistered [12250.556347] Key type ._llcrypt registered [12250.558703] Key type .llcrypt registered [12250.643996] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12250.864382] Key type .llcrypt unregistered [12250.866133] Key type ._llcrypt unregistered [12273.136276] Key type ._llcrypt registered [12273.144417] Key type .llcrypt registered [12273.228469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12273.453379] Key type .llcrypt unregistered [12273.455763] Key type ._llcrypt unregistered [12294.807786] Key type ._llcrypt registered [12294.809913] Key type .llcrypt registered [12294.895826] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12295.111326] Key type .llcrypt unregistered [12295.115944] Key type ._llcrypt unregistered [12316.683066] Key type ._llcrypt registered [12316.685484] Key type .llcrypt registered [12316.766051] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12316.983509] Key type .llcrypt unregistered [12316.986077] Key type ._llcrypt unregistered [12338.354897] Key type ._llcrypt registered [12338.357815] Key type .llcrypt registered [12338.456504] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12338.644785] Key type .llcrypt unregistered [12338.649293] Key type ._llcrypt unregistered [12384.874323] Key type ._llcrypt registered [12384.876026] Key type .llcrypt registered [12384.949682] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12385.155534] Key type .llcrypt unregistered [12385.159167] Key type ._llcrypt unregistered [12406.504651] Key type ._llcrypt registered [12406.507810] Key type .llcrypt registered [12406.612061] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12406.837422] Key type .llcrypt unregistered [12406.840780] Key type ._llcrypt unregistered [12425.975245] Key type ._llcrypt registered [12425.977122] Key type .llcrypt registered [12426.046620] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12426.289318] Key type .llcrypt unregistered [12426.294089] Key type ._llcrypt unregistered [12460.782488] Key type ._llcrypt registered [12460.784592] Key type .llcrypt registered [12460.868210] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [12470.991509] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [12471.427988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12471.443433] alg: No test for adler32 (adler32-zlib) [12472.346966] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [12472.478767] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [12474.104159] Key type lgssc registered [12474.766381] Lustre: Echo OBD driver; http://www.lustre.org/ [12497.774738] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12497.809596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12498.971857] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12498.992069] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12499.046837] Lustre: tst250fs-MDT0000: new disk, initializing [12499.101210] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12499.118905] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12501.212501] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12508.049614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.147442] Lustre: 357327:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12508.201553] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12508.205880] Lustre: Skipped 1 previous similar message [12508.280155] Lustre: tst250fs-MDT0001: new disk, initializing [12508.373793] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12508.401040] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12508.412077] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12511.023487] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12514.232442] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12518.060936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12518.219169] Lustre: tst250fs-OST0000: new disk, initializing [12518.222606] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12518.227677] Lustre: 358928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12518.289199] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12520.721132] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12520.727800] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12520.789117] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12521.577102] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12527.755114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12527.836546] Lustre: tst250fs-OST0001: new disk, initializing [12527.839620] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12527.843830] Lustre: 359781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12527.895692] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12530.904855] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12534.253059] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12534.323210] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12534.328302] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12534.404967] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12549.602716] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12549.609612] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12549.625492] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12550.118986] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12550.128457] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12550.130330] Lustre: Skipped 2 previous similar messages [12550.139982] Lustre: Skipped 3 previous similar messages [12554.591448] Lustre: server umount tst250fs-MDT0000 complete [12556.626546] LustreError: 357322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250863 with bad export cookie 6465513454466792487 [12556.632935] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12556.637761] LustreError: 357322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12556.832870] Lustre: server umount tst250fs-MDT0001 complete [12569.626840] Lustre: server umount tst250fs-OST0000 complete [12581.486562] Lustre: server umount tst250fs-OST0001 complete [12592.029822] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [12596.764145] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [12617.380122] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:55:23 (1781250923) [12618.376566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12627.738405] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [12633.707925] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [12651.007907] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12359 sec ========= 03:55:56 (1781250956) [12652.007580] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:55:57 (1781250957) === [12653.702237] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:55:59 (1781250959) === [12672.943332] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12674.649934] Key type lgssc unregistered [12674.833586] LNet: 368802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12674.840598] LNetError: 368802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12674.853986] LNet: Removed LNI 192.168.203.123@tcp [12675.300158] Key type .llcrypt unregistered [12675.302950] Key type ._llcrypt unregistered