[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491257276 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003274] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008334] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.022834] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027517] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031042] *** VALIDATE tmpfs *** [ 0.032431] *** VALIDATE proc *** [ 0.034248] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038003] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044326] debug: unmapping init [mem 0xffffffffa8859000-0xffffffffa8860fff] [ 0.046189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.050868] ... value mask: 0000ffffffffffff [ 0.051021] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054317] rcu: Hierarchical SRCU implementation. [ 0.056428] smp: Bringing up secondary CPUs ... [ 0.057622] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.061786] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136358] node 0 deferred pages initialised in 69ms [ 0.138286] devtmpfs: initialized [ 0.139261] x86/mm: Memory block size: 128MB [ 0.141670] gcov: version magic: 0x41383552 [ 0.142616] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149347] pinctrl core: initialized pinctrl subsystem [ 0.151172] [ 0.151855] ************************************************************* [ 0.155015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158012] ** ** [ 0.161014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164012] ** ** [ 0.165014] ** This means that this kernel is built to expose internal ** [ 0.169015] ** IOMMU data structures, which may compromise security on ** [ 0.172011] ** your system. ** [ 0.175013] ** ** [ 0.177010] ** If you see this message and you are not debugging the ** [ 0.180014] ** kernel, report this immediately to your vendor! ** [ 0.183013] ** ** [ 0.186013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188012] ************************************************************* [ 0.191702] NET: Registered protocol family 16 [ 0.194437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207020] cpuidle: using governor menu [ 0.208957] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209479] PCI: Using configuration type 1 for base access [ 0.210134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220137] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228033] cryptd: max_cpu_qlen set to 1000 [ 0.231185] ACPI: Added _OSI(Module Device) [ 0.233012] ACPI: Added _OSI(Processor Device) [ 0.234016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241039] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248385] ACPI: Interpreter enabled [ 0.250067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251011] ACPI: Using IOAPIC for interrupt routing [ 0.253101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267351] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281420] acpiphp: Slot [2] registered [ 0.283130] acpiphp: Slot [5] registered [ 0.285177] acpiphp: Slot [6] registered [ 0.286143] acpiphp: Slot [7] registered [ 0.288118] acpiphp: Slot [8] registered [ 0.290137] acpiphp: Slot [9] registered [ 0.291148] acpiphp: Slot [10] registered [ 0.293144] acpiphp: Slot [3] registered [ 0.294112] acpiphp: Slot [4] registered [ 0.296106] acpiphp: Slot [11] registered [ 0.297129] acpiphp: Slot [12] registered [ 0.299088] acpiphp: Slot [13] registered [ 0.300117] acpiphp: Slot [14] registered [ 0.302113] acpiphp: Slot [15] registered [ 0.304116] acpiphp: Slot [16] registered [ 0.306119] acpiphp: Slot [17] registered [ 0.308125] acpiphp: Slot [18] registered [ 0.309102] acpiphp: Slot [19] registered [ 0.310090] acpiphp: Slot [20] registered [ 0.312110] acpiphp: Slot [21] registered [ 0.313141] acpiphp: Slot [22] registered [ 0.314103] acpiphp: Slot [23] registered [ 0.316121] acpiphp: Slot [24] registered [ 0.318129] acpiphp: Slot [25] registered [ 0.319173] acpiphp: Slot [26] registered [ 0.321124] acpiphp: Slot [27] registered [ 0.322120] acpiphp: Slot [28] registered [ 0.324108] acpiphp: Slot [29] registered [ 0.325109] acpiphp: Slot [30] registered [ 0.327182] acpiphp: Slot [31] registered [ 0.328076] PCI host bridge to bus 0000:00 [ 0.330018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346149] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348939] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352522] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362755] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381538] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384869] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391061] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423007] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470157] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.508017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.519812] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.525021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.566822] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.594023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.613016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.622236] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.629015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.637017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.656017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667060] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.679017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.690016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.710025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.720729] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.725424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.726375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.728383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.731267] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.737152] iommu: Default domain type: Passthrough [ 0.739553] SCSI subsystem initialized [ 0.741136] ACPI: bus type USB registered [ 0.743130] usbcore: registered new interface driver usbfs [ 0.745101] usbcore: registered new interface driver hub [ 0.747081] usbcore: registered new device driver usb [ 0.749193] pps_core: LinuxPPS API ver. 1 registered [ 0.751012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754052] PTP clock support registered [ 0.756052] EDAC MC: Ver: 3.0.0 [ 0.758111] PCI: Using ACPI for IRQ routing [ 0.760866] NetLabel: Initializing [ 0.761010] NetLabel: domain hash size = 128 [ 0.763013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.765082] NetLabel: unlabeled traffic allowed by default [ 0.767132] vgaarb: loaded [ 0.769025] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.771014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.778162] clocksource: Switched to clocksource kvm-clock [ 0.884001] VFS: Disk quotas dquot_6.6.0 [ 0.885437] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888086] *** VALIDATE ramfs *** [ 0.889535] *** VALIDATE hugetlbfs *** [ 0.891172] pnp: PnP ACPI init [ 0.893699] pnp: PnP ACPI: found 6 devices [ 0.914052] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917890] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.920324] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.922424] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.926248] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928713] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931933] NET: Registered protocol family 2 [ 0.934908] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.940600] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.945414] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.951411] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.955428] TCP: Hash tables configured (established 65536 bind 65536) [ 0.958812] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.963506] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.966852] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971195] NET: Registered protocol family 1 [ 0.975536] RPC: Registered named UNIX socket transport module. [ 0.978878] RPC: Registered udp transport module. [ 0.981018] RPC: Registered tcp transport module. [ 0.983117] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985809] NET: Registered protocol family 44 [ 0.987682] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990315] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.992292] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.994554] PCI: CLS 0 bytes, default 64 [ 0.996116] Unpacking initramfs... [ 2.397696] debug: unmapping init [mem 0xffff9af0fcc54000-0xffff9af0fffbffff] [ 2.402748] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.405521] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.409219] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.917217] Initialise system trusted keyrings [ 2.919015] Key type blacklist registered [ 2.921567] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.932705] zbud: loaded [ 2.936995] *** VALIDATE nfs *** [ 2.938578] *** VALIDATE nfs4 *** [ 2.940333] pstore: using deflate compression [ 2.943766] Platform Keyring initialized [ 3.055491] NET: Registered protocol family 38 [ 3.057378] Key type asymmetric registered [ 3.058949] Asymmetric key parser 'x509' registered [ 3.060846] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.064379] io scheduler mq-deadline registered [ 3.066511] io scheduler kyber registered [ 3.068645] io scheduler bfq registered [ 3.071502] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.075095] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.078604] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.082126] ACPI: Power Button [PWRF] [ 3.088412] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.095987] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.110906] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.118619] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.136799] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.165158] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.193404] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.198583] Non-volatile memory driver v1.3 [ 3.200339] Linux agpgart interface v0.103 [ 3.233885] virtio_blk virtio1: [vda] 149776 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.237509] vda: detected capacity change from 0 to 76685312 [ 3.254034] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.256847] vdb: detected capacity change from 0 to 1073741824 [ 3.274791] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.278711] vdc: detected capacity change from 0 to 2621440000 [ 3.296619] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.299683] vdd: detected capacity change from 0 to 2621440000 [ 3.322093] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.325861] vde: detected capacity change from 0 to 4294967296 [ 3.340397] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.343805] vdf: detected capacity change from 0 to 4294967296 [ 3.350263] libphy: Fixed MDIO Bus: probed [ 3.357391] usbcore: registered new interface driver usbserial_generic [ 3.359862] usbserial: USB Serial support registered for generic [ 3.362468] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.367471] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.369582] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.372413] mousedev: PS/2 mouse device common for all mice [ 3.375486] rtc_cmos 00:05: RTC can wake from S4 [ 3.378635] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.382913] rtc_cmos 00:05: registered as rtc0 [ 3.385846] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.386189] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.389090] intel_pstate: CPU model not supported [ 3.394520] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.400223] hid: raw HID events driver (C) Jiri Kosina [ 3.402708] usbcore: registered new interface driver usbhid [ 3.404799] usbhid: USB HID core driver [ 3.406576] drop_monitor: Initializing network drop monitor service [ 3.410084] Initializing XFRM netlink socket [ 3.412284] NET: Registered protocol family 10 [ 3.415244] Segment Routing with IPv6 [ 3.416781] NET: Registered protocol family 17 [ 3.419641] mpls_gso: MPLS GSO support [ 3.427424] RAS: Correctable Errors collector initialized. [ 3.429796] AVX version of gcm_enc/dec engaged. [ 3.431512] AES CTR mode by8 optimization enabled [ 3.521495] sched_clock: Marking stable (3521451595, 0)->(4429581771, -908130176) [ 3.525515] registered taskstats version 1 [ 3.527687] Loading compiled-in X.509 certificates [ 3.530411] zswap: loaded using pool lzo/zbud [ 3.560612] Key type big_key registered [ 3.576193] Key type encrypted registered [ 3.577850] ima: No TPM chip found, activating TPM-bypass! [ 3.580117] ima: Allocated hash algorithm: sha1 [ 3.582134] ima: No architecture policies found [ 3.584143] evm: Initialising EVM extended attributes: [ 3.586290] evm: security.selinux [ 3.587536] evm: security.ima [ 3.588720] evm: security.capability [ 3.590275] evm: HMAC attrs: 0x1 [ 3.592921] rtc_cmos 00:05: setting system clock to 2026-09-05 05:28:13 UTC (1788586093) [ 3.599600] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 3.603316] debug: unmapping init [mem 0xffffffffa8582000-0xffffffffa8858fff] [ 3.611113] Write protecting the kernel read-only data: 28672k [ 3.614704] debug: unmapping init [mem 0xffffffffa6c03000-0xffffffffa6dfffff] [ 3.617751] debug: unmapping init [mem 0xffffffffa7514000-0xffffffffa75fffff] [ 3.649723] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.657687] systemd[1]: Detected virtualization kvm. [ 3.659724] systemd[1]: Detected architecture x86-64. [ 3.661866] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.692253] systemd[1]: No hostname configured. [ 3.693687] systemd[1]: Set hostname to . [ 3.695822] random: systemd: uninitialized urandom read (16 bytes read) [ 3.698454] systemd[1]: Initializing machine ID from random generator. [ 3.863483] random: systemd: uninitialized urandom read (16 bytes read) [ 3.866578] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.874791] random: systemd: uninitialized urandom read (16 bytes read) [ 3.877652] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.883602] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.603392] device-mapper: uevent: version 1.0.3 [ 4.605281] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.453153] random: fast init done [ 5.490610] virtio_net virtio0 ens2: renamed from eth0 [ 5.522918] scsi host0: ata_piix [ 5.548553] scsi host1: ata_piix [ 5.556831] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.559994] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.067378] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.098872] random: crng init done [ 10.100354] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.700165] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.929722] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.210629] SELinux: Disabled at runtime. [ 12.271661] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.282463] systemd[1]: Detected virtualization kvm. [ 12.284645] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.826499] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.831446] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.836993] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.842414] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.846960] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.857249] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.861945] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ 13.072505] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.384767] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.705181] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.832120] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.876544] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.936136] EDAC sbridge: Ver: 1.1.2 [ 15.682495] Key type dns_resolver registered [ 16.002295] NFS: Registering the id_resolver key type [ 16.004787] Key type id_resolver registered [ 16.006991] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg408-server login: [ 47.270497] libcfs: loading out-of-tree module taints kernel. [ 47.345415] Key type ._llcrypt registered [ 47.350193] Key type .llcrypt registered [ 47.497866] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 69.574160] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 71.512797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.565577] alg: No test for adler32 (adler32-zlib) [ 73.123291] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 74.268188] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 76.071362] Key type lgssc registered [ 77.874603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.077013] hrtimer: interrupt took 2029817 ns [ 96.338353] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.183202] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 149.428655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.467960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.759469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.815581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.908205] Lustre: lustre-MDT0000: new disk, initializing [ 151.006889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.033820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.517480] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.681981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.856238] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.911583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.915605] Lustre: Skipped 1 previous similar message [ 170.049670] Lustre: lustre-MDT0001: new disk, initializing [ 170.117751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.156325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.166785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.896593] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.686786] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.037543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.493895] Lustre: lustre-OST0000: new disk, initializing [ 189.503087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.515144] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.630474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.637134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.651806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.715907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.560328] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.686902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.833502] Lustre: lustre-OST0001: new disk, initializing [ 213.837968] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.843468] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.956229] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.821748] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.774892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.791439] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.827798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 232.417113] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.405704] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.506095] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.480573] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing yml_node [ 256.320090] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 258.978814] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 261.653771] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 263.369386] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 01:32:31 EDT 2026 [ 281.343898] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 283.147780] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.510151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.521858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.539079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.100724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.114692] Lustre: Skipped 6 previous similar messages [ 304.512544] Lustre: server umount lustre-MDT0000 complete [ 308.709753] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586398 with bad export cookie 1026040019425650533 [ 308.712735] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.729529] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 309.225587] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.233437] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.249029] Lustre: Skipped 4 previous similar messages [ 313.248114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.265443] Lustre: Skipped 2 previous similar messages [ 315.244385] Lustre: server umount lustre-MDT0001 complete [ 319.653261] Lustre: server umount lustre-OST0000 complete [ 323.297203] Lustre: server umount lustre-OST0001 complete [ 329.780667] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 339.093972] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 376.963559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.326722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 377.380241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.529599] Lustre: lustre-MDT0000: new disk, initializing [ 377.669257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.691844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 382.546657] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 393.343848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.515969] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 393.570786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 393.575902] Lustre: Skipped 1 previous similar message [ 393.660545] Lustre: lustre-MDT0001: new disk, initializing [ 393.759471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.806412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 393.822247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 398.848844] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 410.415522] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 416.540862] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 423.181789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.506495] Lustre: lustre-OST0000: new disk, initializing [ 423.510306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 423.515871] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 423.618560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.823767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 424.852914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 424.929820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 429.695577] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 439.121664] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 443.880833] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.125342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.793438] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.112701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.651917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.652154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.673537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.678285] Lustre: Skipped 1 previous similar message [ 457.447147] Lustre: server umount lustre-OST0000 complete [ 465.383652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 465.397200] LustreError: Skipped 1 previous similar message [ 465.406311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.422031] Lustre: Skipped 1 previous similar message [ 465.431747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.589842] Lustre: server umount lustre-MDT0000 complete [ 471.016128] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.032697] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 471.291540] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586561 with bad export cookie 1026040019425652066 [ 471.298890] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 471.304616] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 471.591311] Lustre: server umount lustre-MDT0001 complete [ 478.556344] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:36:06 (1788586566) [ 497.821258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.200513] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.288362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.584396] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 503.777254] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.907767] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.071812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.123275] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 524.382286] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 531.617284] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 534.298077] Lustre: Failing over lustre-MDT0000 [ 534.581085] Lustre: server umount lustre-MDT0000 complete [ 536.380187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.390260] LustreError: 19912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.400292] Lustre: Skipped 2 previous similar messages [ 536.445537] LustreError: 19912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 546.784890] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.810877] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 552.543099] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788586626/real 1788586626] req@ffff9af0432f0e00 x1875468525657856/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788586642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 552.562210] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.404629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.854537] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.893683] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 563.114819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.122954] Lustre: Skipped 1 previous similar message [ 568.300958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 568.302073] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 568.357359] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 568.376781] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 579.117532] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 586.720506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.724168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 586.735962] Lustre: Skipped 1 previous similar message [ 586.750902] Lustre: Skipped 2 previous similar messages [ 590.466265] Lustre: server umount lustre-MDT0000 complete [ 594.708879] LustreError: 20652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586684 with bad export cookie 1026040019425653480 [ 594.718368] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.732187] LustreError: 20652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.152533] Lustre: server umount lustre-MDT0001 complete [ 604.742732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.555051] LustreError: 23254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.582900] LustreError: 23254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 612.118351] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 621.462216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.254914] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 637.743523] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 645.214395] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 647.679587] Lustre: Failing over lustre-MDT0000 [ 647.870121] Lustre: server umount lustre-MDT0000 complete [ 651.747300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.768035] Lustre: Skipped 1 previous similar message [ 667.105583] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788586741/real 1788586741] req@ffff9af17fa6d500 x1875468525690496/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788586757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 667.151161] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.311759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.227374] LustreError: 23254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.253137] LustreError: 23254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 677.354473] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe3d39f623337692 [ 677.379503] LustreError: 25259:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.108@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff9af043798700) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 677.403597] Lustre: MGC192.168.204.108@tcp: Connection restored to 0@lo (at 0@lo) [ 677.419867] Lustre: Skipped 1 previous similar message [ 677.688612] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 682.991660] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 683.008824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 683.047743] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 683.057888] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 693.986605] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 702.439111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.442083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.468345] Lustre: Skipped 1 previous similar message [ 704.785456] Lustre: server umount lustre-MDT0000 complete [ 709.232933] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586799 with bad export cookie 1026040019425654418 [ 709.251156] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.283987] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 709.759328] Lustre: server umount lustre-MDT0001 complete [ 720.802786] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:40:08 (1788586808) [ 743.332600] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 756.242933] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 808.227894] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 824.030862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.368398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 824.448929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 824.577677] Lustre: lustre-MDT0000: new disk, initializing [ 824.666427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.675895] Lustre: Skipped 3 previous similar messages [ 824.693696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 830.742454] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 844.702628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.894590] Lustre: 29652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 845.199394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 852.428982] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 866.161100] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 874.357743] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 882.379399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 882.789042] Lustre: 31624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 884.651195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 884.688121] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 884.788104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 884.807893] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 888.943295] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 899.572484] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 906.514856] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 930.793107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.797738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.803312] Lustre: Skipped 1 previous similar message [ 930.814195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.815189] Lustre: Skipped 2 previous similar messages [ 935.440735] Lustre: server umount lustre-OST0000 complete [ 946.904947] Lustre: server umount lustre-MDT0000 complete [ 950.242968] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.257239] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 952.077864] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587041 with bad export cookie 1026040019425654831 [ 952.086555] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.092686] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 952.901657] Lustre: server umount lustre-MDT0001 complete [ 967.760978] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 971.586566] Key type lgssc unregistered [ 971.971834] LNet: 33605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.979913] LNetError: 33605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.044531] LNet: Removed LNI 192.168.204.108@tcp [ 974.394344] Key type .llcrypt unregistered [ 974.403715] Key type ._llcrypt unregistered [ 1001.705171] Key type ._llcrypt registered [ 1001.710590] Key type .llcrypt registered [ 1001.866237] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1003.596813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.647631] alg: No test for adler32 (adler32-zlib) [ 1004.797547] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1005.099413] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1006.743224] Key type lgssc registered [ 1007.916607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1017.445968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1017.483948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.170700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.505059] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1029.601354] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.617206] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1032.448282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.923103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1038.240145] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1049.066301] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1056.290992] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1064.427365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1064.714257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.892977] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1078.069610] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1080.309934] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1097.424880] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:46:24 (1788587184) [ 1099.579495] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1101.614101] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1109.756560] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:46:37 (1788587197) [ 1119.502442] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:46:46 (1788587206) [ 1120.758206] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1131.040995] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1131.049741] Lustre: Skipped 2 previous similar messages [ 1133.488174] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:47:00 (1788587220) [ 1136.313780] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1136.325937] Lustre: Skipped 2 previous similar messages [ 1145.552490] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1145.558748] Lustre: Skipped 4 previous similar messages [ 1162.337249] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1162.345407] Lustre: Skipped 2 previous similar messages [ 1165.577457] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:47:33 (1788587253) [ 1168.857319] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1173.227583] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.036929] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:47:51 (1788587271) [ 1203.128754] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1203.134327] Lustre: Skipped 7 previous similar messages [ 1205.581263] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:48:12 (1788587292) [ 1224.678141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1224.691085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.708563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1226.727869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1226.732324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1226.751863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.878558] Lustre: server umount lustre-OST0000 complete [ 1236.972797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.995859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.009728] Lustre: Skipped 2 previous similar messages [ 1239.996297] Lustre: server umount lustre-MDT0000 complete [ 1242.091961] LustreError: 40559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.124926] LustreError: 40559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1245.389969] LustreError: 35750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587335 with bad export cookie 4732360994230698895 [ 1245.392714] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.407813] LustreError: 35750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1246.320188] Lustre: server umount lustre-MDT0001 complete [ 1259.956431] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1263.969130] Key type lgssc unregistered [ 1264.374887] LNet: 42883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.379955] LNetError: 42883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.394911] LNet: Removed LNI 192.168.204.108@tcp [ 1265.865555] Key type .llcrypt unregistered [ 1265.870581] Key type ._llcrypt unregistered [ 1291.480858] Key type ._llcrypt registered [ 1291.484604] Key type .llcrypt registered [ 1291.562662] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 1309.218694] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1310.901215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.970321] alg: No test for adler32 (adler32-zlib) [ 1312.275701] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1312.681215] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1314.391366] Key type lgssc registered [ 1315.578903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1361.654984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1361.690853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.064116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1363.098858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1363.167481] Lustre: lustre-MDT0000: new disk, initializing [ 1363.260511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.313285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1369.433729] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1384.159816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.397290] Lustre: 46042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1384.573937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1384.583894] Lustre: Skipped 1 previous similar message [ 1384.767871] Lustre: lustre-MDT0001: new disk, initializing [ 1384.904049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.945345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1384.956176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1389.958873] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1402.316530] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1409.979555] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1418.632121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1418.838849] Lustre: lustre-OST0000: new disk, initializing [ 1418.842547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1418.851514] Lustre: 48015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1418.976665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.438149] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1427.601187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1427.621360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1427.779214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1437.661356] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1443.736609] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1444.050286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1449.660550] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1449.939701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1453.540159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.547949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1453.576031] LustreError: Skipped 1 previous similar message [ 1453.577592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.588694] Lustre: Skipped 1 previous similar message [ 1458.535855] Lustre: server umount lustre-OST0000 complete [ 1463.798295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.810647] Lustre: Skipped 1 previous similar message [ 1463.812569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.825170] Lustre: Skipped 1 previous similar message [ 1469.046604] Lustre: server umount lustre-MDT0000 complete [ 1473.337935] LustreError: 46033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587563 with bad export cookie 478139970001884501 [ 1473.345629] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.361183] LustreError: 46033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1473.820739] Lustre: server umount lustre-MDT0001 complete [ 1485.016178] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:52:52 (1788587572) [ 1493.665033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.310164] LustreError: 50475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.338272] LustreError: 50475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1494.393228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.177303] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1499.628753] LustreError: 50476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.738761] LustreError: 50475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.626432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.860781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1510.108469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.062118] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1524.117535] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1530.946683] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1539.146631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1539.679340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.284975] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1556.105210] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1571.532613] LustreError: 53025:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1572.635742] Lustre: Modifying parameter general.timeout=1 in log params [ 1573.696457] Lustre: Modifying parameter general.timeout=2 in log params [ 1575.707842] Lustre: Modifying parameter general.timeout=4 in log params [ 1575.710239] Lustre: Skipped 1 previous similar message [ 1580.734414] Lustre: Modifying parameter general.timeout=9 in log params [ 1580.740542] Lustre: Skipped 4 previous similar messages [ 1589.133385] Lustre: Modifying parameter general.timeout=17 in log params [ 1589.136511] Lustre: Skipped 7 previous similar messages [ 1605.826500] Lustre: Modifying parameter general.timeout=33 in log params [ 1605.830414] Lustre: Skipped 15 previous similar messages [ 1638.572299] Lustre: Modifying parameter general.timeout=62 in log params [ 1638.575640] Lustre: Skipped 28 previous similar messages [ 1703.655574] Lustre: Modifying parameter general.timeout=123 in log params [ 1703.667610] Lustre: Skipped 60 previous similar messages [ 1723.269902] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:56:50 (1788587810) [ 1726.508675] Lustre: 59981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1728.672701] Lustre: 60079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1730.828721] Lustre: 60177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1736.827393] Lustre: 60488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1736.849575] Lustre: 60488:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1767.057384] Lustre: 61861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1767.089683] LustreError: 61861:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1772.865975] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:57:39 (1788587859) [ 1790.438286] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.473201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.481903] Lustre: Skipped 2 previous similar messages [ 1795.559951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.573018] Lustre: Skipped 2 previous similar messages [ 1796.260376] Lustre: server umount lustre-MDT0000 complete [ 1799.944553] LustreError: 50455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587889 with bad export cookie 478139970001885390 [ 1799.958859] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.962095] LustreError: 50455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1800.389383] Lustre: server umount lustre-MDT0001 complete [ 1815.221443] Lustre: server umount lustre-OST0000 complete [ 1848.471457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.696375] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1848.718912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1848.724160] Lustre: Skipped 29 previous similar messages [ 1848.878608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.748469] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1864.840972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.964734] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1865.011782] Lustre: 64396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1871.069456] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1881.256943] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1887.561408] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1894.873696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.072827] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1895.386932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.391439] Lustre: Skipped 1 previous similar message [ 1902.854372] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1911.598678] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1913.484906] Lustre: 66964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1920.993441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1921.006723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.024989] Lustre: Skipped 2 previous similar messages [ 1921.037451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1926.384902] Lustre: server umount lustre-OST0000 complete [ 1931.263173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.274792] Lustre: Skipped 2 previous similar messages [ 1938.060616] Lustre: server umount lustre-MDT0000 complete [ 1941.471878] LustreError: 64408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.493698] LustreError: 64408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1942.472192] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588032 with bad export cookie 478139970001888589 [ 1942.482810] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.497667] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1942.877142] Lustre: server umount lustre-MDT0001 complete [ 1955.527909] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1958.704732] Key type lgssc unregistered [ 1959.000612] LNet: 68169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.012885] LNetError: 68169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.047647] LNet: Removed LNI 192.168.204.108@tcp [ 1960.107307] Key type .llcrypt unregistered [ 1960.109707] Key type ._llcrypt unregistered [ 1986.439227] Key type ._llcrypt registered [ 1986.445256] Key type .llcrypt registered [ 1986.557892] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1987.530194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1987.615134] alg: No test for adler32 (adler32-zlib) [ 1988.773762] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1988.985382] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1990.673045] Key type lgssc registered [ 1991.789646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.061404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.090167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.608850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.205964] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2011.121197] LustreError: 69595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.144084] LustreError: 69595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2016.229789] LustreError: 69596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.507716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.080765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.847994] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2030.184966] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2036.188370] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.267245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2043.768457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.716364] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2058.748666] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2067.231941] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2076.644261] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.656687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.662066] Lustre: Skipped 1 previous similar message [ 2076.678667] Lustre: Skipped 2 previous similar messages [ 2081.770244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.230785] Lustre: server umount lustre-MDT0000 complete [ 2086.225737] LustreError: 69575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588176 with bad export cookie 17933059088091278602 [ 2086.226512] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.243744] LustreError: 69575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2086.565429] Lustre: server umount lustre-MDT0001 complete [ 2101.219041] Lustre: server umount lustre-OST0000 complete [ 2129.823800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.195641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2130.243680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2130.641711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.910577] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2144.904189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.070994] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2145.115744] Lustre: 74468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2145.736737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.182232] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2159.715957] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.219673] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2174.825496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2175.198227] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2175.729548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.956744] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2191.638935] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2194.205137] Lustre: 77046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2196.449901] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2203.109087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.124326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.137438] Lustre: Skipped 1 previous similar message [ 2203.142683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.150075] Lustre: Skipped 2 previous similar messages [ 2203.625932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2208.737309] LustreError: 76320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.825255] Lustre: server umount lustre-OST0000 complete [ 2217.441792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2217.455292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.471730] Lustre: Skipped 1 previous similar message [ 2217.480210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.489416] Lustre: Skipped 1 previous similar message [ 2220.301956] Lustre: server umount lustre-MDT0000 complete [ 2224.075648] LustreError: 75214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588313 with bad export cookie 17933059088091279442 [ 2224.078262] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2224.083025] LustreError: 75214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2224.408176] Lustre: server umount lustre-MDT0001 complete [ 2239.222967] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2242.994674] Key type lgssc unregistered [ 2243.433628] LNet: 78255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.448867] LNetError: 78255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.474721] LNet: Removed LNI 192.168.204.108@tcp [ 2244.765141] Key type .llcrypt unregistered [ 2244.767960] Key type ._llcrypt unregistered [ 2276.370400] Key type ._llcrypt registered [ 2276.374955] Key type .llcrypt registered [ 2276.608319] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2278.075029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.118331] alg: No test for adler32 (adler32-zlib) [ 2279.164838] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2279.459375] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2281.103244] Key type lgssc registered [ 2282.452652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2291.579332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2291.612876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.391121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.261325] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2303.970444] LustreError: 79666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.986876] LustreError: 79666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2308.293778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.104462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2314.577753] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2323.960910] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2331.406602] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.629458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2340.381853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.213477] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2357.413945] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2374.494478] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2375.485155] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2377.417895] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2377.424408] Lustre: Skipped 1 previous similar message [ 2379.662282] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2379.676365] Lustre: Skipped 1 previous similar message [ 2383.990425] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2383.996142] Lustre: Skipped 3 previous similar messages [ 2392.101426] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2392.105095] Lustre: Skipped 6 previous similar messages [ 2408.700514] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2408.705408] Lustre: Skipped 15 previous similar messages [ 2441.732931] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2441.739895] Lustre: Skipped 29 previous similar messages [ 2492.120680] Lustre: 87805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2495.620188] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:09:43 (1788588583) [ 2497.525815] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2499.859394] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:09:47 (1788588587) [ 2507.234931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.237842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.253984] LustreError: Skipped 1 previous similar message [ 2507.257315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.298898] Lustre: Skipped 1 previous similar message [ 2511.910103] Lustre: server umount lustre-OST0000 complete [ 2517.481865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.519265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.529143] Lustre: Skipped 2 previous similar messages [ 2521.959790] Lustre: server umount lustre-MDT0000 complete [ 2522.594796] LustreError: 80927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.617785] LustreError: 80927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2526.252604] LustreError: 82279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588616 with bad export cookie 6405683664806518549 [ 2526.259291] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.267457] LustreError: 82279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2526.550439] Lustre: server umount lustre-MDT0001 complete [ 2539.571963] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2542.320452] Key type lgssc unregistered [ 2542.642867] LNet: 89052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.664223] LNetError: 89052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.689656] LNet: Removed LNI 192.168.204.108@tcp [ 2543.504931] Key type .llcrypt unregistered [ 2543.507505] Key type ._llcrypt unregistered [ 2547.510490] Key type ._llcrypt registered [ 2547.512516] Key type .llcrypt registered [ 2547.595573] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2552.973336] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules [ 2554.052924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2554.167094] alg: No test for adler32 (adler32-zlib) [ 2555.604771] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2555.996036] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2557.847633] Key type lgssc registered [ 2559.601515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2561.311713] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2581.386478] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2583.523123] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2592.015494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2592.027982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.742181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.680309] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2604.001648] LustreError: 91277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.039442] LustreError: 91277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2609.130073] LustreError: 91276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.464425] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:11:38 (1788588698) [ 2614.245389] LustreError: 91277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.713440] LustreError: 91308:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2616.737445] LustreError: 91308:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2616.743707] LustreError: 91308:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2617.055272] Lustre: server umount lustre-MDT0000 complete [ 2634.119219] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2637.292963] Key type lgssc unregistered [ 2637.617748] LNet: 92801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.637302] LNetError: 92801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.703031] LNet: Removed LNI 192.168.204.108@tcp [ 2639.773769] Key type .llcrypt unregistered [ 2639.778272] Key type ._llcrypt unregistered [ 2661.106042] Key type ._llcrypt registered [ 2661.114059] Key type .llcrypt registered [ 2661.260753] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2662.963112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2663.044946] alg: No test for adler32 (adler32-zlib) [ 2664.106134] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2664.493421] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2666.263228] Key type lgssc registered [ 2667.638451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2676.868073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2676.900638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.592302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.152241] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2688.993274] LustreError: 94016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.021088] LustreError: 94016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2694.119779] LustreError: 94017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.957176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.647021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.785560] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2711.217523] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.639375] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2724.613089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2725.289247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.350957] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2742.051310] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2811.324637] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:14:58 (1788588898) [ 2813.061108] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2815.418311] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:15:02 (1788588902) [ 2823.136504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2823.145636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.157095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.696047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.700186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.725649] Lustre: Skipped 1 previous similar message [ 2827.181944] Lustre: server umount lustre-MDT0000 complete [ 2830.820242] LustreError: 94016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.834676] LustreError: 94016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2831.360802] LustreError: 93995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588921 with bad export cookie 15680912215901972396 [ 2831.364664] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.372814] LustreError: 93995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2831.706693] Lustre: server umount lustre-MDT0001 complete [ 2846.459826] Lustre: server umount lustre-OST0000 complete [ 2860.071924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.566663] LustreError: 98332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.732703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.341600] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2866.146141] LustreError: 98333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.176804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.361307] LustreError: 98333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.377881] LustreError: 98333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2876.068935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2881.400509] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2889.023758] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2896.778893] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2914.284307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.631536] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2914.647845] LustreError: 98322:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2914.693545] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2914.704996] LustreError: 100419:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2914.716438] LustreError: 100419:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2914.736574] LustreError: 100419:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2914.820624] Lustre: server umount lustre-OST0000 complete [ 2914.823738] LustreError: 100419:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2924.371138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.618703] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2924.631440] LustreError: 98321:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2924.665189] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2924.675730] LustreError: 100887:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2924.694974] LustreError: 100887:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2924.705121] LustreError: 100887:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2924.790863] Lustre: server umount lustre-OST0000 complete [ 2924.795626] LustreError: 100887:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2936.043562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.245065] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2936.253671] Lustre: Found index 0 for lustre-OST0000, updating log [ 2936.258950] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2936.345491] Lustre: lustre-OST0000: new disk, initializing [ 2936.354639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2936.369940] Lustre: 101422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2936.789386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.300383] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2948.932158] Lustre: Failing over lustre-OST0000 [ 2949.172579] Lustre: server umount lustre-OST0000 complete [ 2956.259765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.266587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.284070] Lustre: Skipped 1 previous similar message [ 2956.298580] Lustre: Skipped 3 previous similar messages [ 2960.001582] Lustre: server umount lustre-MDT0000 complete [ 2961.378984] LustreError: 98332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.410028] LustreError: 98332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2964.734324] LustreError: 99073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589054 with bad export cookie 15680912215901975140 [ 2964.744048] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.747641] LustreError: 99073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2964.753353] LustreError: 99065:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2964.775877] LustreError: 99065:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2974.414566] Lustre: server umount lustre-MDT0001 complete [ 2986.441702] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:17:53 (1788589073) [ 3007.274806] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3019.072826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.570443] LustreError: 103771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.764596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.293346] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3035.005115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.529276] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3045.140504] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3053.011702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.888917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.903639] Lustre: Skipped 1 previous similar message [ 3053.930994] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3061.033571] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3061.059588] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3061.060736] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3061.728461] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3072.106161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3072.215868] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3072.395261] Lustre: lustre-OST0001: new disk, initializing [ 3072.400381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3072.410197] Lustre: 106090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3078.762533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3078.772972] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3078.878402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3080.950637] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3090.513084] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3099.804503] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3104.258449] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [ 3104.748260] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.768976] Lustre: Skipped 1 previous similar message [ 3109.373401] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [ 3109.381890] Lustre: Skipped 3 previous similar messages [ 3118.047491] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3118.174199] Lustre: server umount lustre-MDT0001 complete [ 3119.614054] LustreError: 106333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.635549] LustreError: 106333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3130.619743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.760363] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3130.766810] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3130.853673] Lustre: 103760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3131.823321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3131.832878] Lustre: Skipped 1 previous similar message [ 3132.326345] LustreError: 107631:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3132.331379] LustreError: 107631:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3132.337933] LustreError: 103762:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3132.434094] LustreError: 107637:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.108@tcp: cfg command failed: rc = -17 [ 3132.444760] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3132.444760] [ 3137.006631] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3137.601930] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3150.854955] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:20:37 (1788589237) [ 3170.757854] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3189.097826] Lustre: 109710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3213.377438] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3218.006354] Lustre: 110844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3315.694762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3315.709897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.721060] Lustre: Skipped 1 previous similar message [ 3315.734995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.741640] Lustre: Skipped 7 previous similar messages [ 3317.816526] Lustre: server umount lustre-MDT0000 complete [ 3321.323072] LustreError: 106333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.364890] LustreError: 106333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 3322.301499] LustreError: 103753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589412 with bad export cookie 15680912215901976141 [ 3322.313373] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.319513] LustreError: 103753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3322.971660] Lustre: server umount lustre-MDT0001 complete [ 3337.959206] Lustre: server umount lustre-OST0000 complete [ 3352.336389] Lustre: server umount lustre-OST0001 complete [ 3358.349863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3369.575254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3379.735667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3393.467219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3415.897213] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3429.026748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.055396] Lustre: lustre-MDT0000: reset Object Index mappings [ 3429.697466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.564475] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3446.003173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.037882] Lustre: lustre-MDT0001: reset Object Index mappings [ 3452.522253] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3456.060155] Lustre: 115951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3464.303616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.306825] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3473.018849] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3475.442317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3484.448758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.301853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3490.341279] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3492.394628] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3502.515510] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3506.953477] Lustre: 117831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3571.921415] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:27:38 (1788589658) [ 3595.233297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3595.247487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.267268] Lustre: Skipped 3 previous similar messages [ 3595.273234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.276635] Lustre: Skipped 4 previous similar messages [ 3601.059647] Lustre: server umount lustre-MDT0000 complete [ 3602.914296] LustreError: 114803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.926518] LustreError: 114803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3604.661249] LustreError: 114784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589694 with bad export cookie 15680912215902023034 [ 3604.666528] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.686433] LustreError: 114784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3611.222082] Lustre: server umount lustre-MDT0001 complete [ 3622.268832] Lustre: server umount lustre-OST0000 complete [ 3632.779452] Lustre: server umount lustre-OST0001 complete [ 3640.179216] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 3651.077505] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3693.924104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.188419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3694.249835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3694.376385] Lustre: lustre-MDT0000: new disk, initializing [ 3694.448554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.457048] Lustre: Skipped 3 previous similar messages [ 3694.475118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3700.331967] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3708.966871] Lustre: server umount lustre-MDT0000 complete [ 3720.490540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3720.976117] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3721.011480] Lustre: 122104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3721.032583] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3727.058630] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3732.392825] Lustre: server umount lustre-MDT0000 complete [ 3741.473207] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:30:29 (1788589829) [ 3743.339792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3745.141483] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:30:32 (1788589832) [ 3764.851745] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3776.290114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.169977] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3791.331355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.503751] Lustre: 124155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3791.586204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3791.590971] Lustre: Skipped 1 previous similar message [ 3791.713429] Lustre: lustre-MDT0001: new disk, initializing [ 3791.930163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3791.946809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3796.956148] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3801.775775] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3809.192690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3809.592148] Lustre: lustre-OST0000: new disk, initializing [ 3809.599996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3809.612495] Lustre: 125736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3811.311206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3811.331542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3811.484563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3816.066142] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3826.996990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3827.216942] Lustre: lustre-OST0001: new disk, initializing [ 3827.219834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3827.225029] Lustre: 126608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3829.088093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3829.101270] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3829.257450] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3834.473373] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3845.396527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3850.100913] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4277.455816] Lustre: 133032:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4277.480098] Lustre: 133032:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4277.490990] Lustre: 133032:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4277.498415] Lustre: 133032:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4277.513713] Lustre: 133032:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4277.520317] Lustre: 133032:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4610.752804] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:44:57 (1788590697) [ 4627.952834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.965213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.979314] Lustre: Skipped 10 previous similar messages [ 4627.980969] Lustre: Skipped 8 previous similar messages [ 4629.420365] Lustre: server umount lustre-MDT0000 complete [ 4633.085133] LustreError: 126327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.110253] LustreError: 126327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4633.596646] LustreError: 124147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590723 with bad export cookie 15680912215902093188 [ 4633.597892] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4633.615103] LustreError: 124147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4638.185856] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.206001] Lustre: Skipped 1 previous similar message [ 4638.211323] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4638.232409] Lustre: Skipped 2 previous similar messages [ 4640.363487] Lustre: server umount lustre-MDT0001 complete [ 4651.480644] Lustre: server umount lustre-OST0000 complete [ 4662.163520] Lustre: server umount lustre-OST0001 complete [ 4671.185447] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4683.209637] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4724.770088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.034759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4725.100748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4725.249135] Lustre: lustre-MDT0000: new disk, initializing [ 4725.354463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.357055] Lustre: Skipped 5 previous similar messages [ 4725.394920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4731.071441] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4743.133918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.327616] Lustre: 136102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4743.436422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4743.448915] Lustre: Skipped 1 previous similar message [ 4743.584166] Lustre: lustre-MDT0001: new disk, initializing [ 4743.792189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4743.808106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4748.857986] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4759.410267] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4765.975391] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4773.401171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4773.739697] Lustre: lustre-OST0000: new disk, initializing [ 4773.743086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4773.748989] Lustre: 138071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4775.350518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4775.366637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4775.531464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4780.656862] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4792.003784] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4805.823399] Lustre: lustre-MDD0000: changelog on [ 4808.726988] Lustre: lustre-MDD0001: changelog on [ 6975.632701] Lustre: lustre-MDD0001: changelog off [ 6978.107919] Lustre: 139526:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6979.095810] Lustre: lustre-MDD0000: changelog off [ 6987.234871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.238215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.249560] LustreError: Skipped 1 previous similar message [ 6987.251161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.275814] Lustre: Skipped 1 previous similar message [ 6990.314807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.321988] Lustre: Skipped 2 previous similar messages [ 6995.454167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6995.464032] Lustre: Skipped 1 previous similar message [ 7000.032036] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7000.289221] Lustre: server umount lustre-OST0000 complete [ 7005.667959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.681261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.689547] Lustre: Skipped 1 previous similar message [ 7011.280365] Lustre: server umount lustre-MDT0000 complete [ 7015.905611] LustreError: 136980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.930673] LustreError: 136980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7016.020898] LustreError: 136097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593105 with bad export cookie 15680912215902924613 [ 7016.029430] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.036583] LustreError: 136097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7016.409311] Lustre: server umount lustre-MDT0001 complete [ 7029.081973] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 7032.317956] Key type lgssc unregistered [ 7032.663223] LNet: 147258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7032.675711] LNetError: 147258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7032.696272] LNet: Removed LNI 192.168.204.108@tcp [ 7033.648182] Key type .llcrypt unregistered [ 7033.656243] Key type ._llcrypt unregistered [ 7056.924852] Key type ._llcrypt registered [ 7056.927643] Key type .llcrypt registered [ 7057.051664] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7075.888898] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7077.471768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7077.767875] alg: No test for adler32 (adler32-zlib) [ 7079.012552] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7079.342939] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 7081.105937] Key type lgssc registered [ 7082.656782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7122.790845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7122.832988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7124.141309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7124.184535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7124.277396] Lustre: lustre-MDT0000: new disk, initializing [ 7124.357501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.384468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7129.659142] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7140.463926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.602187] Lustre: 150457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7140.699379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7140.702780] Lustre: Skipped 1 previous similar message [ 7140.809791] Lustre: lustre-MDT0001: new disk, initializing [ 7140.895604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7140.932369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7140.949564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7145.200531] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7156.227103] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7162.673219] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7170.417162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.699645] Lustre: lustre-OST0000: new disk, initializing [ 7170.703145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7170.710100] Lustre: 152424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7170.818400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.778286] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7178.810695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7178.823488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7178.927588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7187.596752] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7194.156634] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_module obdecho/obdecho [ 7196.115517] LustreError: 153427:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7196.127592] LustreError: 153427:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7198.071759] LustreError: 153524:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7212.512314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7212.516792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.528968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7215.079792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.079871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7215.101978] Lustre: Skipped 1 previous similar message [ 7217.333972] Lustre: server umount lustre-MDT0000 complete [ 7221.855883] LustreError: 150448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593311 with bad export cookie 7445376560513935069 [ 7221.858975] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.867323] LustreError: 150448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7222.587386] Lustre: server umount lustre-MDT0001 complete [ 7237.728704] Lustre: server umount lustre-OST0000 complete [ 7262.205541] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7275.124934] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7313.556680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.786844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7313.825619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7313.894761] Lustre: lustre-MDT0000: new disk, initializing [ 7313.999592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.019706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7319.522696] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7332.760178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7332.900268] Lustre: 157242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7332.945382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7332.947925] Lustre: Skipped 1 previous similar message [ 7333.030420] Lustre: lustre-MDT0001: new disk, initializing [ 7333.216978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7333.273737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7333.289752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7338.602057] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7350.767778] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7357.512424] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7365.014885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7365.323606] Lustre: lustre-OST0000: new disk, initializing [ 7365.329618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7365.337108] Lustre: 159210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7365.442189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.852802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7372.865055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7373.024077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7373.764413] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7385.575193] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7391.227934] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7391.496486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7396.928688] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7397.232313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7404.014809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7404.025448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.030561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7404.036675] Lustre: Skipped 2 previous similar messages [ 7404.047645] Lustre: Skipped 1 previous similar message [ 7405.854569] Lustre: server umount lustre-OST0000 complete [ 7414.245490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.248534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7414.268226] Lustre: Skipped 1 previous similar message [ 7414.288751] Lustre: Skipped 1 previous similar message [ 7416.774144] Lustre: server umount lustre-MDT0000 complete [ 7419.360742] LustreError: 157250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7419.381645] LustreError: 157250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7421.209729] LustreError: 157234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593511 with bad export cookie 7445376560513936021 [ 7421.219824] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.225586] LustreError: 157234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7421.713658] Lustre: server umount lustre-MDT0001 complete [ 7426.261379] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:31:53 (1788593513) [ 7428.242710] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7430.653911] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:31:57 (1788593517) [ 7439.857399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.322600] LustreError: 161625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7440.426183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7445.481023] LustreError: 161626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7445.521974] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7450.595032] LustreError: 161625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.763741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7458.498160] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7467.720733] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7478.807653] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7487.541040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7488.022583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.027429] Lustre: Skipped 1 previous similar message [ 7495.331369] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7505.180364] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7520.740562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.742378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7520.746600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.755502] Lustre: Skipped 1 previous similar message [ 7520.778992] LustreError: Skipped 1 previous similar message [ 7523.271793] Lustre: server umount lustre-MDT0000 complete [ 7525.862158] LustreError: 163471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.890566] LustreError: 163471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7527.365267] LustreError: 161607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593617 with bad export cookie 7445376560513936910 [ 7527.368355] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.382331] LustreError: 161607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7530.980753] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7531.024188] Lustre: Skipped 1 previous similar message [ 7531.037489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7531.050667] Lustre: Skipped 3 previous similar messages [ 7533.750690] Lustre: server umount lustre-MDT0001 complete [ 7540.584601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7554.236112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.385610] LustreError: 165490:0:(llog.c:1655:llog_backup()) MGC192.168.204.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7554.390676] Lustre: 165490:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7557.602161] LustreError: 165495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.611573] LustreError: 165495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7578.086259] LustreError: 165497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.117700] LustreError: 165497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7613.409204] LustreError: 165495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.427889] LustreError: 165495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 7614.943342] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7614.964120] Lustre: 165490:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7655.935570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7660.747753] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7661.033954] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7670.477416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.296536] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x67534a88d4740744 [ 7672.324559] Lustre: MGC192.168.204.108@tcp: Connection restored to 0@lo (at 0@lo) [ 7677.647519] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7677.939367] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7685.934489] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7687.784154] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7694.689601] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7696.987792] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7718.885319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.901811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7718.912174] Lustre: Skipped 2 previous similar messages [ 7719.185408] Lustre: server umount lustre-MDT0000 complete [ 7722.917389] LustreError: 161606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593812 with bad export cookie 7445376560513943364 [ 7722.922072] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7722.931446] LustreError: 161606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7723.411291] Lustre: server umount lustre-MDT0001 complete [ 7737.950950] Lustre: server umount lustre-OST0000 complete [ 7760.973589] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7772.881294] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7812.544440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.783281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7812.823934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7812.965550] Lustre: lustre-MDT0000: new disk, initializing [ 7813.049350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7813.054726] Lustre: Skipped 1 previous similar message [ 7813.079390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7817.892560] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7829.081249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.245917] Lustre: 170597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7829.608580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7834.635504] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7845.489705] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7851.730991] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7859.106176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7859.443975] Lustre: lustre-OST0000: new disk, initializing [ 7859.449227] Lustre: Skipped 1 previous similar message [ 7859.458896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7859.463816] Lustre: Skipped 2 previous similar messages [ 7859.470130] Lustre: 172566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7865.932224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7865.950609] Lustre: Skipped 1 previous similar message [ 7865.956782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7866.109233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7867.136255] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7877.514129] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7882.143536] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7882.461908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7886.746365] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7886.918693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7891.938155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.946339] Lustre: Skipped 2 previous similar messages [ 7891.948624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7891.957106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.964279] LustreError: Skipped 1 previous similar message [ 7894.827820] Lustre: server umount lustre-OST0000 complete [ 7901.153537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7905.418571] Lustre: server umount lustre-MDT0000 complete [ 7907.296953] LustreError: 171738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.320271] LustreError: 171738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 7910.148350] LustreError: 170588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594000 with bad export cookie 7445376560513960206 [ 7910.167709] LustreError: 170588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7910.181364] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.621938] Lustre: server umount lustre-MDT0001 complete [ 7920.318147] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:40:08 (1788594008) [ 7928.219094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.736372] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7942.256611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7946.391864] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7953.508584] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7958.891188] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7966.252385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7972.801844] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7980.422639] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7989.970839] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:41:17 (1788594077) [ 7994.344750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7994.349732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.355130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.355140] Lustre: Skipped 4 previous similar messages [ 7994.380678] Lustre: Skipped 4 previous similar messages [ 8000.091114] Lustre: server umount lustre-OST0000 complete [ 8018.399252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8018.564234] Lustre: server umount lustre-MDT0000 complete [ 8022.838757] LustreError: 174959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594112 with bad export cookie 7445376560513961088 [ 8022.846573] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.847981] LustreError: 174959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8023.300891] Lustre: server umount lustre-MDT0001 complete [ 8034.017631] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 8036.863377] Key type lgssc unregistered [ 8037.173622] LNet: 178706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8037.181166] LNetError: 178706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8037.194623] LNet: Removed LNI 192.168.204.108@tcp [ 8038.195568] Key type .llcrypt unregistered [ 8038.200172] Key type ._llcrypt unregistered [ 8060.034290] Key type ._llcrypt registered [ 8060.037189] Key type .llcrypt registered [ 8060.168137] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8061.446632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8061.893171] alg: No test for adler32 (adler32-zlib) [ 8063.025465] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 8063.384883] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 8065.127559] Key type lgssc registered [ 8066.568354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8074.019288] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8074.039225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.632279] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8094.752196] LustreError: 179947:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 8099.871616] LustreError: 179947:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8099.887611] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8099.898163] LustreError: MGC192.168.204.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8099.914797] LustreError: 179947:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8099.928854] LustreError: 179947:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8099.938881] LustreError: 179947:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8100.010365] Lustre: server umount lustre-OST0000 complete [ 8100.015407] LustreError: 179947:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8109.867127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.383812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.582729] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8121.824780] LustreError: 180445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.842112] LustreError: 180445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8125.919798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.254317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8131.241156] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8139.871768] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8146.309991] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8155.690372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8156.007419] LustreError: 182265:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8156.029866] Lustre: 182265:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8156.353667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.331986] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8174.502521] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8177.123064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.134344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.241652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.249810] Lustre: Skipped 4 previous similar messages [ 8183.360755] Lustre: server umount lustre-MDT0000 complete [ 8187.342617] LustreError: 183026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594277 with bad export cookie 14880094603352862619 [ 8187.367267] LustreError: 183026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8187.378025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.381774] LustreError: 180442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8187.392955] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.402522] Lustre: Skipped 2 previous similar messages [ 8187.433431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8187.707699] Lustre: server umount lustre-MDT0001 complete [ 8194.739776] Lustre: server umount lustre-OST0000 complete [ 8207.683137] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 8218.969140] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8261.030307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.337385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8261.369123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8261.496937] Lustre: lustre-MDT0000: new disk, initializing [ 8261.622644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.641755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8266.545617] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8278.661847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.803256] Lustre: 185787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8278.893726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8278.900283] Lustre: Skipped 1 previous similar message [ 8278.994166] Lustre: lustre-MDT0001: new disk, initializing [ 8279.131075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8279.186660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8279.208774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8284.408570] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8295.475592] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8301.835521] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8310.474869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8310.918736] Lustre: lustre-OST0000: new disk, initializing [ 8310.929640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8310.948465] Lustre: 187760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8311.092290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8317.511692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8317.520371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8317.670301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8318.470340] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8328.893932] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8334.818443] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8335.089761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8340.540374] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8340.759256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8343.521111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.525489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.530993] LustreError: Skipped 1 previous similar message [ 8343.534478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8343.552633] Lustre: Skipped 1 previous similar message [ 8343.571073] Lustre: Skipped 1 previous similar message [ 8348.663429] LustreError: 187745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8348.686540] LustreError: 187745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8348.818925] Lustre: server umount lustre-OST0000 complete [ 8353.771957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.789279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.803479] Lustre: Skipped 1 previous similar message [ 8359.713994] Lustre: server umount lustre-MDT0000 complete [ 8363.901598] LustreError: 185781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594453 with bad export cookie 14880094603352863340 [ 8363.909272] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8363.919098] LustreError: 185781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8364.269348] Lustre: server umount lustre-MDT0001 complete [ 8375.426719] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:47:42 (1788594462) [ 8393.105939] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 8396.332346] Key type lgssc unregistered [ 8396.613282] LNet: 190704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8396.622736] LNetError: 190704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8396.656598] LNet: Removed LNI 192.168.204.108@tcp [ 8397.545403] Key type .llcrypt unregistered [ 8397.548068] Key type ._llcrypt unregistered [ 8417.132496] Key type ._llcrypt registered [ 8417.133848] Key type .llcrypt registered [ 8417.336412] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8419.251199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8419.299838] alg: No test for adler32 (adler32-zlib) [ 8420.435198] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 8420.682617] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 8422.343310] Key type lgssc registered [ 8424.014936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8433.857524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8433.873583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8435.379653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8440.959870] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8445.924639] LustreError: 191920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8445.949347] LustreError: 191920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8451.065895] LustreError: 191921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8451.707543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.312739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8458.022108] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8467.333906] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8476.003243] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8486.948538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.197718] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8487.201224] LustreError: 193786:0:(llog.c:1696:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8487.208889] Lustre: 193786:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8487.311738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.906524] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8506.532464] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8510.946366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.948792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.961889] Lustre: Skipped 2 previous similar messages [ 8510.988660] Lustre: Skipped 2 previous similar messages [ 8515.483806] Lustre: server umount lustre-MDT0000 complete [ 8516.069556] LustreError: 191920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8516.097830] LustreError: 191920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8520.787422] LustreError: 193813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594610 with bad export cookie 10488975793985506444 [ 8520.790199] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8520.804159] LustreError: 193813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8521.213142] Lustre: server umount lustre-MDT0001 complete [ 8536.397211] Lustre: server umount lustre-OST0000 complete [ 8549.356225] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 8564.370709] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8614.226332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.609874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8614.660857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8614.781838] Lustre: lustre-MDT0000: new disk, initializing [ 8614.886349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8614.917450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8621.101812] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8635.330223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.505872] Lustre: 197335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8635.550089] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8635.552049] Lustre: Skipped 1 previous similar message [ 8635.555282] LustreError: 198120:0:(llog.c:1696:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8635.561703] LustreError: 198120:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8635.565725] Lustre: 198120:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8635.580717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8635.585209] Lustre: Skipped 1 previous similar message [ 8635.687590] Lustre: lustre-MDT0001: new disk, initializing [ 8635.852091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8635.923215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8635.935794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8642.423404] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8654.547955] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8660.921807] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8671.099266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8671.338294] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8671.340100] Lustre: Skipped 1 previous similar message [ 8671.344820] LustreError: 199282:0:(llog.c:1696:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8671.362412] LustreError: 199282:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8671.370268] Lustre: 199282:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8671.393052] Lustre: 199282:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8671.409715] Lustre: lustre-OST0000: new disk, initializing [ 8671.415355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8671.421469] Lustre: 199307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8671.537388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8680.013866] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8680.637958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8680.653712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8680.842081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8692.992387] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8699.937784] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8700.238470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8706.318505] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8706.729283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8711.650302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.655757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8711.684876] Lustre: Skipped 1 previous similar message [ 8715.854792] Lustre: server umount lustre-OST0000 complete [ 8721.890933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8721.929153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8721.944494] Lustre: Skipped 2 previous similar messages [ 8727.010147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8727.015855] Lustre: Skipped 1 previous similar message [ 8728.099349] Lustre: server umount lustre-MDT0000 complete [ 8731.112406] LustreError: 197347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8731.154341] LustreError: 197347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8733.240972] LustreError: 197327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594823 with bad export cookie 10488975793985507165 [ 8733.250952] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8733.252113] LustreError: 197327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8734.070794] Lustre: server umount lustre-MDT0001 complete [ 8744.994606] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:53:52 (1788594832) [ 8765.188403] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8777.353148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.968778] LustreError: 202437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8778.057375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8784.172201] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8787.428712] LustreError: 202438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.458236] LustreError: 202438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8794.744341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.854535] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8804.692813] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8812.566407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8813.021349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8813.043192] Lustre: Skipped 1 previous similar message [ 8821.697906] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8832.671974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8832.954551] Lustre: lustre-OST0001: new disk, initializing [ 8832.959567] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8832.969575] Lustre: 204748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8841.421821] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8842.923297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8842.942994] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8843.092192] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8855.656401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8865.758639] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8868.393249] Lustre: DEBUG MARKER: ADD OST3 [ 8874.680976] loop: module loaded [ 8874.815948] loop0: detected capacity change from 0 to 204800000 [ 8875.230913] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8875.242637] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8875.264423] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8883.327255] Lustre: DEBUG MARKER: STOP OST3 [ 8883.557540] loop0: detected capacity change from 0 to 204800000 [ 8891.331200] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.598411] Lustre: lustre-OST0003: new disk, initializing [ 8891.604647] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8891.622101] Lustre: 206791:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8891.847632] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8891.860526] Lustre: Skipped 1 previous similar message [ 8893.064883] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8893.273611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8893.288695] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8893.349485] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8895.021279] Lustre: Failing over lustre-OST0003 [ 8895.135193] LustreError: 206894:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8895.202860] Lustre: server umount lustre-OST0003 complete [ 8898.554333] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8898.559404] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8898.571603] Lustre: Skipped 1 previous similar message [ 8898.572904] LustreError: 205182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.572912] LustreError: 205182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8900.446525] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8909.379870] Lustre: DEBUG MARKER: START OST3 again [ 8918.141738] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8918.492504] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8919.038818] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8920.402693] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8920.408696] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8927.394667] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8951.912120] Lustre: Failing over lustre-OST0003 [ 8952.035360] Lustre: server umount lustre-OST0003 complete [ 8954.375458] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.398662] Lustre: Skipped 1 previous similar message [ 8954.407327] LustreError: 205182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8954.441505] LustreError: 205182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8974.307773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8974.319238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.334380] Lustre: Skipped 1 previous similar message [ 8974.339298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8978.687402] Lustre: server umount lustre-MDT0000 complete [ 8982.900862] LustreError: 202418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595072 with bad export cookie 10488975793985508047 [ 8982.927059] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.928717] LustreError: 202418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8985.062902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8985.072024] Lustre: Skipped 4 previous similar messages [ 8997.345575] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8997.909581] Lustre: server umount lustre-MDT0001 complete [ 9003.484286] Lustre: server umount lustre-OST0000 complete [ 9008.581843] Lustre: server umount lustre-OST0001 complete [ 9015.519696] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 9023.777363] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9069.457371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9069.805588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9069.851086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9069.964149] Lustre: lustre-MDT0000: new disk, initializing [ 9070.101331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9070.111816] Lustre: Skipped 1 previous similar message [ 9070.137559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9075.632418] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9087.325542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9087.519820] Lustre: 212214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9087.593098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9087.597362] Lustre: Skipped 1 previous similar message [ 9087.700787] Lustre: lustre-MDT0001: new disk, initializing [ 9087.859433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9087.868203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9093.250529] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9103.174432] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9109.488041] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9119.059311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9119.314742] Lustre: 214183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9120.407182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9120.418462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9120.587848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9129.968767] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9143.119704] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9148.659746] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9148.998858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9153.466555] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9153.798349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9156.576577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9156.583637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.588518] LustreError: Skipped 1 previous similar message [ 9156.595028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9156.615794] Lustre: Skipped 6 previous similar messages [ 9156.636718] Lustre: Skipped 7 previous similar messages [ 9170.399982] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9170.572425] Lustre: server umount lustre-OST0000 complete [ 9190.368785] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9190.550364] Lustre: server umount lustre-MDT0000 complete [ 9192.417560] LustreError: 212219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9192.444923] LustreError: 212219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 9195.493906] LustreError: 212205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595285 with bad export cookie 10488975793985510343 [ 9195.495293] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9195.511847] LustreError: 212205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9196.107142] Lustre: server umount lustre-MDT0001 complete [ 9200.265400] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 04:01:27 (1788595287) [ 9218.261888] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 9230.628357] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9269.823343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9270.035247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9270.054355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9270.060727] Lustre: Skipped 1 previous similar message [ 9270.140306] Lustre: lustre-MDT0000: new disk, initializing [ 9270.144060] Lustre: Skipped 1 previous similar message [ 9270.225660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9275.014432] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9287.063354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.297580] Lustre: 218816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9287.676089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9292.877614] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9304.128500] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9311.649899] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9320.201527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9320.574699] Lustre: 220786:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9322.247690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9322.377485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9327.776309] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9339.030276] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9344.668925] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9344.986693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9350.430807] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9350.705452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9353.186574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9353.187586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9353.201112] LustreError: Skipped 1 previous similar message [ 9353.206666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9353.206675] Lustre: Skipped 10 previous similar messages [ 9353.273648] Lustre: Skipped 3 previous similar messages [ 9367.009755] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9367.308165] Lustre: server umount lustre-OST0000 complete [ 9377.508754] Lustre: server umount lustre-MDT0000 complete [ 9378.789359] LustreError: 218829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9378.800037] LustreError: 218829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9381.429216] LustreError: 219623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595471 with bad export cookie 10488975793985511225 [ 9381.430748] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9381.446190] LustreError: 219623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9381.870939] Lustre: server umount lustre-MDT0001 complete [ 9390.208029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.545286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9390.548173] Lustre: Skipped 5 previous similar messages [ 9395.753491] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9406.158292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.657806] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9420.536522] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9426.906163] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9435.009713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9441.507267] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9451.053835] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9462.245159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9462.251044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.257695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9462.273747] Lustre: Skipped 3 previous similar messages [ 9462.294117] Lustre: Skipped 8 previous similar messages [ 9467.533619] Lustre: server umount lustre-OST0000 complete [ 9477.777697] Lustre: server umount lustre-MDT0000 complete [ 9481.507797] LustreError: 222984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595571 with bad export cookie 10488975793985512107 [ 9481.520616] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9481.522494] LustreError: 222984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9482.045833] Lustre: server umount lustre-MDT0001 complete [ 9591.484775] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 9594.306313] Key type lgssc unregistered [ 9594.565321] LNet: 226667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9594.582902] LNetError: 226667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9594.604909] LNet: Removed LNI 192.168.204.108@tcp [ 9595.535217] Key type .llcrypt unregistered [ 9595.537848] Key type ._llcrypt unregistered [ 9620.920748] Key type ._llcrypt registered [ 9620.923264] Key type .llcrypt registered [ 9620.997698] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 9637.235413] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9638.078262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9638.092217] alg: No test for adler32 (adler32-zlib) [ 9639.156792] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 9639.407550] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 9641.064254] Key type lgssc registered [ 9642.135023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9685.967879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9686.007986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.363795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9687.390747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9687.482330] Lustre: lustre-MDT0000: new disk, initializing [ 9687.564179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9687.590717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9693.419397] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9704.926713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.091258] Lustre: 230020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9705.166727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9705.171874] Lustre: Skipped 1 previous similar message [ 9705.245366] Lustre: lustre-MDT0001: new disk, initializing [ 9705.383763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9705.450025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9705.470130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9711.948938] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9723.955353] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9729.796333] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9738.183132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.471685] Lustre: lustre-OST0000: new disk, initializing [ 9738.477147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9738.486864] Lustre: 231988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9738.598089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9746.065431] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9748.557899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9748.573019] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9748.652237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9755.828262] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9761.566866] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9761.742628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9766.266105] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9766.493658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9769.444357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9769.445152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9769.460668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9769.465812] Lustre: Skipped 1 previous similar message [ 9774.563174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9774.574967] Lustre: Skipped 2 previous similar messages [ 9774.844592] Lustre: server umount lustre-OST0000 complete [ 9779.682898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.690151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.700573] Lustre: Skipped 1 previous similar message [ 9784.801875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9784.811146] Lustre: Skipped 2 previous similar messages [ 9785.207649] Lustre: server umount lustre-MDT0000 complete [ 9788.969147] LustreError: 230010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595878 with bad export cookie 3656128040939646176 [ 9788.981045] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9789.389496] Lustre: server umount lustre-MDT0001 complete [ 9812.236509] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9823.731044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.306657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9829.114167] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9829.365749] LustreError: 234922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9829.380076] LustreError: 234922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9834.465920] LustreError: 234923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.500928] LustreError: 234922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9838.256697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.825191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9843.689792] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9846.795412] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9852.787577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9853.102176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9859.495035] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9868.911937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9869.084729] Lustre: lustre-OST0001: new disk, initializing [ 9869.089217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9869.095401] Lustre: 237230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9869.187215] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9875.099290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9875.120792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9875.224600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9876.247316] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9888.003471] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9895.558817] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9908.252315] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 04:13:15 (1788595995) [ 9910.755597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9910.764601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.776093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9916.765112] Lustre: server umount lustre-MDT0000 complete [ 9920.603293] LustreError: 234902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596010 with bad export cookie 3656128040939647058 [ 9920.608806] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9920.622641] LustreError: 234902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9921.029581] Lustre: server umount lustre-MDT0001 complete [ 9935.765590] Lustre: server umount lustre-OST0000 complete [ 9937.695921] Lustre: 228242:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788596011/real 1788596011] req@ffff9af170369500 x1875478555242112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788596027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9937.720459] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9937.734202] Lustre: Skipped 3 previous similar messages [ 9940.329120] Lustre: server umount lustre-OST0001 complete [ 9948.302189] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 9960.352997] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10000.790321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.135337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10001.166369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10001.267598] Lustre: lustre-MDT0000: new disk, initializing [10001.345137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10001.361666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10006.167423] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10017.198499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.346185] Lustre: 241175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10017.404964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10017.410273] Lustre: Skipped 1 previous similar message [10017.517270] Lustre: lustre-MDT0001: new disk, initializing [10017.640240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10017.651393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10021.737294] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10031.143555] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10037.855218] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10046.198835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10046.478613] Lustre: lustre-OST0000: new disk, initializing [10046.482554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10046.488836] Lustre: 243145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10048.227819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10048.246835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10048.398798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10053.917841] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10064.710654] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10070.161706] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10070.486407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10075.682674] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10075.927830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10079.213107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10079.214220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.217913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10079.217930] Lustre: Skipped 8 previous similar messages [10079.225715] LustreError: Skipped 1 previous similar message [10079.263636] Lustre: Skipped 1 previous similar message [10084.333236] LustreError: 243131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.439245] Lustre: server umount lustre-OST0000 complete [10089.445179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10089.458572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10095.124678] Lustre: server umount lustre-MDT0000 complete [10099.276356] LustreError: 241167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596189 with bad export cookie 3656128040939649193 [10099.283285] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.286370] LustreError: 241167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10099.708613] Lustre: server umount lustre-MDT0001 complete [10108.959607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.353562] LustreError: 245363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10109.371547] LustreError: 245363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10109.434919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10109.440943] Lustre: Skipped 2 previous similar messages [10114.896977] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10119.651500] LustreError: 245364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.673880] LustreError: 245364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10124.569158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.541645] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10139.401437] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10145.490809] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10153.534299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10162.345879] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10170.807801] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10318.307609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10318.308564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.315279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10318.330322] Lustre: Skipped 2 previous similar messages [10318.340715] Lustre: Skipped 6 previous similar messages [10321.501917] Lustre: server umount lustre-OST0000 complete [10331.685471] Lustre: server umount lustre-MDT0000 complete [10333.668438] LustreError: 247012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10333.693525] LustreError: 247012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10335.971029] LustreError: 245345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596425 with bad export cookie 3656128040939650075 [10335.987280] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10335.990480] LustreError: 245345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10336.376045] Lustre: server umount lustre-MDT0001 complete [10440.560301] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [10443.804490] Key type lgssc unregistered [10444.191569] LNet: 248973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10444.194810] LNetError: 248973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10444.213177] LNet: Removed LNI 192.168.204.108@tcp [10445.124230] Key type .llcrypt unregistered [10445.126627] Key type ._llcrypt unregistered [10474.089607] Key type ._llcrypt registered [10474.092197] Key type .llcrypt registered [10474.211336] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [10491.853029] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10493.173783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10493.382888] alg: No test for adler32 (adler32-zlib) [10494.635446] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [10495.107232] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [10496.895692] Key type lgssc registered [10498.604954] Lustre: Echo OBD driver; http://www.lustre.org/ [10557.195844] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10569.078061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10569.115181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10570.432192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10570.476162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10570.554973] Lustre: lustre-MDT0000: new disk, initializing [10570.664677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10570.689451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10575.493725] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10587.516281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10587.686285] Lustre: 253065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10587.767103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10587.775788] Lustre: Skipped 1 previous similar message [10587.886801] Lustre: lustre-MDT0001: new disk, initializing [10588.065337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10588.151075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10588.170552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10593.566635] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10598.639311] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10606.165555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10606.507416] Lustre: lustre-OST0000: new disk, initializing [10606.513293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10606.519622] Lustre: 254696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10606.618959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10611.853950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10611.882664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10612.051356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10614.892518] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10626.459583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10626.602501] Lustre: lustre-OST0001: new disk, initializing [10626.605883] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10626.611436] Lustre: 255568:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10626.703796] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10632.264430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10632.269931] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10632.357971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10632.437397] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10645.336976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10655.856380] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10670.053129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10670.059182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10670.079922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10673.639778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10673.658248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10673.686148] Lustre: Skipped 1 previous similar message [10676.285757] Lustre: server umount lustre-MDT0000 complete [10680.333890] LustreError: 254695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596770 with bad export cookie 923069755254639063 [10680.337182] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10680.350374] LustreError: 254695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10680.687179] Lustre: server umount lustre-MDT0001 complete [10694.902383] Lustre: server umount lustre-OST0000 complete [10708.663512] Lustre: server umount lustre-OST0001 complete [10725.211492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10736.564589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10745.701367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10756.744945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10778.694630] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10790.030272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.046965] Lustre: lustre-MDT0000: reset Object Index mappings [10790.825595] LustreError: 260581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.847938] LustreError: 260581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10790.978205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10796.000082] LustreError: 260582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10796.567886] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10800.098692] LustreError: 260581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.218495] LustreError: 260582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.967302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10805.996878] Lustre: lustre-MDT0001: reset Object Index mappings [10806.546089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10812.074834] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10816.039816] Lustre: 261724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10823.968111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10824.614804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10831.806721] LustreError: 262077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10832.045518] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10835.972827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10841.064813] LustreError: 262078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10841.094719] LustreError: 262078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10842.371913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10847.758334] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10850.677715] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10860.230075] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10864.799072] Lustre: 263597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10873.315596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.321515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10873.333973] Lustre: Skipped 1 previous similar message [10873.349452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10873.349463] Lustre: Skipped 3 previous similar messages [10878.441846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10878.459411] Lustre: Skipped 6 previous similar messages [10883.557891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.562719] Lustre: Skipped 2 previous similar messages [10885.087209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10885.457597] Lustre: server umount lustre-MDT0000 complete [10888.678690] LustreError: 262098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.723134] LustreError: 262098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10890.557705] LustreError: 260561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596980 with bad export cookie 923069755254645853 [10890.563702] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10890.582144] LustreError: 260561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10890.993393] Lustre: server umount lustre-MDT0001 complete [10907.128817] Lustre: server umount lustre-OST0000 complete [10909.151221] Lustre: 250547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788596983/real 1788596983] req@ffff9af16e356d80 x1875479451800448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788596999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10909.206812] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10909.220699] Lustre: Skipped 1 previous similar message [10911.295853] Lustre: server umount lustre-OST0001 complete [10936.669664] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:30:23 (1788597023) [10957.523402] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [10970.609954] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11017.007753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11017.294984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11017.359861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11017.487510] Lustre: lustre-MDT0000: new disk, initializing [11017.560513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11017.565603] Lustre: Skipped 1 previous similar message [11017.578538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11022.895542] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11035.174975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.312583] Lustre: 267858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11035.392637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11035.396724] Lustre: Skipped 1 previous similar message [11035.483233] Lustre: lustre-MDT0001: new disk, initializing [11035.667961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11035.681217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11040.721189] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11052.350285] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11059.027131] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11066.550931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11066.942650] Lustre: lustre-OST0000: new disk, initializing [11066.947899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11066.966661] Lustre: 269826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11068.561831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11068.573685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11068.712811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11075.082937] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11086.724559] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11093.245586] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11093.598147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11099.453342] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11099.704410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11104.739857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11104.753563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.772353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11108.475182] Lustre: server umount lustre-OST0000 complete [11114.987121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.026578] Lustre: Skipped 2 previous similar messages [11119.874553] Lustre: server umount lustre-MDT0000 complete [11120.098060] LustreError: 267869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.139283] LustreError: 267869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11124.678073] LustreError: 267850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597214 with bad export cookie 923069755254647946 [11124.686104] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11124.691182] LustreError: 267850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11125.336682] Lustre: server umount lustre-MDT0001 complete [11150.278577] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11163.710340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11164.338692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11164.343230] Lustre: Skipped 2 previous similar messages [11169.438145] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11178.952555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.168765] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11187.626161] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11194.877993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11203.222591] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11215.828203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11216.034874] Lustre: lustre-OST0001: new disk, initializing [11216.039607] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11216.052314] Lustre: 275074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11218.055646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11218.066805] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11218.185963] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11223.308598] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11235.010971] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11239.284492] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11245.231975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11245.260422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11256.288456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11256.294815] LustreError: Skipped 1 previous similar message [11256.301135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11256.326342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11256.338937] Lustre: Skipped 3 previous similar messages [11262.407567] Lustre: server umount lustre-MDT0000 complete [11264.488127] LustreError: 272762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.520315] LustreError: 272762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11266.770890] LustreError: 275073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597356 with bad export cookie 923069755254648828 [11266.778966] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11266.788293] LustreError: 275073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11267.446778] Lustre: server umount lustre-MDT0001 complete [11281.393780] Lustre: server umount lustre-OST0000 complete [11285.343727] Lustre: 250547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788597359/real 1788597359] req@ffff9af16a12c700 x1875479451915776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788597375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11286.442081] Lustre: server umount lustre-OST0001 complete [11307.414754] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11319.880547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11320.346098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11320.350804] Lustre: Skipped 3 previous similar messages [11326.111465] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11336.080123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.937511] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11345.803568] Lustre: 279562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11354.926783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11363.835177] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11365.801297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11374.880371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11380.235728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11383.613430] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11392.660463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11397.108263] Lustre: 281442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11404.128732] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11404.345082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11419.945892] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:38:27 (1788597507) [11436.522131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.538944] Lustre: Skipped 7 previous similar messages [11436.540279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11436.556206] Lustre: Skipped 7 previous similar messages [11441.799611] Lustre: server umount lustre-MDT0000 complete [11445.912638] LustreError: 278402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597535 with bad export cookie 923069755254655457 [11445.930596] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11445.932739] LustreError: 278402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11446.756863] LustreError: 279938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.792460] LustreError: 279938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11452.408841] Lustre: server umount lustre-MDT0001 complete [11462.613560] Lustre: server umount lustre-OST0000 complete [11467.641617] Lustre: server umount lustre-OST0001 complete [11475.909459] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [11487.167359] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11544.072339] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11558.122833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.546838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11558.581217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11558.743390] Lustre: lustre-MDT0000: new disk, initializing [11558.907682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11564.489700] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11580.456585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11580.656157] Lustre: 285973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11580.953547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11580.963385] Lustre: Skipped 4 previous similar messages [11581.032740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11586.758803] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11593.004223] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11600.669965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11601.047952] Lustre: 287608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11602.650318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11609.586523] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11623.739204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11624.062873] Lustre: 288484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11625.802565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11625.812877] Lustre: Skipped 1 previous similar message [11625.959759] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11633.160889] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11645.615922] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11670.438537] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:42:37 (1788597757) [11673.065894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11673.079882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.095475] Lustre: Skipped 2 previous similar messages [11673.104813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11673.109984] Lustre: Skipped 8 previous similar messages [11673.334527] Lustre: server umount lustre-MDT0000 complete [11677.655174] LustreError: 285964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597767 with bad export cookie 923069755254663171 [11677.666458] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11677.676965] LustreError: 285964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11678.202902] Lustre: server umount lustre-MDT0001 complete [11692.471660] Lustre: server umount lustre-OST0000 complete [11707.637030] Lustre: server umount lustre-OST0001 complete [11719.052431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.619205] LustreError: 290731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.642995] LustreError: 290731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11725.784105] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11737.214919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.018678] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11753.140689] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11760.423643] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11769.101467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11777.111569] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11787.381329] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11813.407203] loop1: detected capacity change from 0 to 204800000 [11813.933792] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11813.951995] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11813.964557] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11821.289834] loop1: detected capacity change from 0 to 204800000 [11830.345306] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11830.549787] LustreError: 290720:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs20928: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11830.560428] LustreError: 290720:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs20928 [11830.566467] LustreError: 290720:0:(mgs_handler.c:517:mgs_target_reg()) fs20928-MDT0001: Error checking target index: rc = -13 [11830.573947] LustreError: fs20928-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11830.581927] LustreError: 294452:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11830.586805] LustreError: 294452:0:(tgt_mount.c:2132:server_put_super()) no obd fs20928-MDT0001 [11830.601396] LustreError: 294452:0:(tgt_mount.c:116:server_deregister_mount()) fs20928-MDT0001 not registered [11830.683100] Lustre: server umount fs20928-MDT0001 complete [11830.695316] LustreError: 294452:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11840.983142] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11841.139944] Lustre: Setting parameter fs20928-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs20928-MDT0001 [11841.175131] Lustre: Skipped 1 previous similar message [11841.292617] Lustre: srv-fs20928-MDT0001: No data found on store. Initialize space. [11841.302925] Lustre: Skipped 4 previous similar messages [11841.482294] Lustre: fs20928-MDT0001: new disk, initializing [11841.487307] Lustre: Skipped 3 previous similar messages [11841.645359] Lustre: srv-fs20928-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11841.654798] Lustre: 294919:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20928-MDT0001-mdtlov: can't create new llog: rc = -115 [11841.675576] LustreError: 294919:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20928-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11847.004237] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11854.670098] Lustre: server umount fs20928-MDT0001 complete [11860.972438] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11870.505880] loop1: detected capacity change from 0 to 204800000 [11877.878892] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.448157] Lustre: 296490:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20928-MDT0001-mdtlov: can't create new llog: rc = -115 [11878.455499] LustreError: 296490:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20928-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11884.561592] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11890.600666] Lustre: server umount fs20928-MDT0001 complete [11906.073151] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.8@tcp (stopping) [11906.531408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11906.562933] LustreError: Skipped 1 previous similar message [11906.567468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11906.589827] Lustre: Skipped 3 previous similar messages [11908.858852] Lustre: server umount lustre-MDT0000 complete [11919.121983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11919.223658] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11924.982758] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11925.787668] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11937.610947] loop1: detected capacity change from 0 to 204800000 [11944.840145] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.536105] Lustre: 298937:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20928-MDT0001-mdtlov: can't create new llog: rc = -115 [11945.541786] LustreError: 298937:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20928-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11951.487873] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11957.956766] Lustre: server umount fs20928-MDT0001 complete [11982.817617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11987.903616] Lustre: server umount lustre-OST0000 complete [11997.892878] Lustre: server umount lustre-MDT0000 complete [12001.921834] LustreError: 290711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598091 with bad export cookie 923069755254665852 [12001.929050] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12001.932953] LustreError: 290711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12002.599102] Lustre: server umount lustre-MDT0001 complete [12014.294947] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [12017.412499] Key type lgssc unregistered [12017.769398] LNet: 301280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12017.791914] LNetError: 301280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12017.825037] LNet: Removed LNI 192.168.204.108@tcp [12018.866266] Key type .llcrypt unregistered [12018.871792] Key type ._llcrypt unregistered [12039.310189] Key type ._llcrypt registered [12039.315040] Key type .llcrypt registered [12039.424621] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [12040.425767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12040.600598] alg: No test for adler32 (adler32-zlib) [12041.656306] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [12042.003774] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [12043.648377] Key type lgssc registered [12045.086426] Lustre: Echo OBD driver; http://www.lustre.org/ [12054.492067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12054.525751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.079219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12061.923971] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12065.264549] LustreError: 302551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.289817] LustreError: 302551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12070.370294] LustreError: 302552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.526963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.020695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12080.686502] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12091.470651] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12099.256739] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12107.622719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12108.112748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12115.852795] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12118.410228] LustreError: 304383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12123.633731] LustreError: 304385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12125.675250] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12128.449659] LustreError: 304854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12128.485334] LustreError: 304854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12139.009788] LustreError: 304383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12139.041384] LustreError: 304383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12148.773790] loop1: detected capacity change from 0 to 204800000 [12149.053600] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12149.062699] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12149.080921] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12156.392718] LustreError: 304854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.413659] LustreError: 304854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [12157.241720] loop1: detected capacity change from 0 to 204800000 [12165.108559] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12165.377418] Lustre: fs14932-OST0000: new disk, initializing [12165.383783] Lustre: srv-fs14932-OST0000: No data found on store. Initialize space. [12165.394825] Lustre: 306293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12165.562860] Lustre: fs14932-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12175.304992] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12185.513524] Lustre: server umount fs14932-OST0000 complete [12190.195034] LustreError: 304384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.240367] LustreError: 304384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [12197.356450] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12201.865606] loop1: detected capacity change from 0 to 204800000 [12202.486124] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12209.121017] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12209.498685] Lustre: fs14932-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12217.779477] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12233.408814] loop2: detected capacity change from 0 to 204800000 [12234.100246] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12234.127820] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12234.156080] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12244.411630] loop2: detected capacity change from 0 to 204800000 [12253.869689] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12253.965287] LustreError: fs14932-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12253.975369] LustreError: fs14932-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12253.980681] LustreError: 309336:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [12253.988742] LustreError: 309336:0:(tgt_mount.c:2132:server_put_super()) no obd fs14932-OST0001 [12253.993587] LustreError: 309336:0:(tgt_mount.c:116:server_deregister_mount()) fs14932-OST0001 not registered [12254.040594] Lustre: server umount fs14932-OST0001 complete [12254.043246] LustreError: 309336:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12256.772774] LustreError: 304384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.800459] LustreError: 304384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12268.085826] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12268.354176] Lustre: fs14932-OST0001: new disk, initializing [12268.357603] Lustre: srv-fs14932-OST0001: No data found on store. Initialize space. [12268.371143] Lustre: 309887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12268.529035] Lustre: fs14932-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12277.667608] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12288.955868] Lustre: server umount fs14932-OST0000 complete [12300.918422] Lustre: server umount fs14932-OST0001 complete [12317.427565] loop1: detected capacity change from 0 to 204800000 [12325.406700] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.701542] Lustre: fs14932-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12333.867037] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12343.239317] loop2: detected capacity change from 0 to 204800000 [12350.972604] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12351.180771] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12356.086147] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12360.582661] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12374.137448] Lustre: server umount fs14932-OST0000 complete [12386.798932] LustreError: 304383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.818377] LustreError: 304383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 76 previous similar messages [12387.414889] Lustre: server umount fs14932-OST0001 complete [12405.220532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12405.228457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.253856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12407.265951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12407.272539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12407.324790] Lustre: Skipped 1 previous similar message [12412.390494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12412.401084] Lustre: Skipped 1 previous similar message [12417.510671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12417.516709] Lustre: Skipped 1 previous similar message [12419.039612] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12419.346178] Lustre: server umount lustre-OST0000 complete [12427.238444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12427.248871] LustreError: Skipped 1 previous similar message [12427.259316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.281262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12431.026538] Lustre: server umount lustre-MDT0000 complete [12437.310685] LustreError: 303296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598527 with bad export cookie 3384758782614803913 [12437.322033] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12437.329451] LustreError: 303296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12437.896603] Lustre: server umount lustre-MDT0001 complete [12453.051451] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [12457.074779] Key type lgssc unregistered [12457.550431] LNet: 315816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12457.581442] LNetError: 315816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12457.625371] LNet: Removed LNI 192.168.204.108@tcp [12459.036354] Key type .llcrypt unregistered [12459.043852] Key type ._llcrypt unregistered [12501.677726] Key type ._llcrypt registered [12501.680040] Key type .llcrypt registered [12501.846239] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [12521.488468] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [12522.955561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12522.982638] alg: No test for adler32 (adler32-zlib) [12524.209297] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [12524.543430] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [12526.255594] Key type lgssc registered [12527.974304] Lustre: Echo OBD driver; http://www.lustre.org/ [12577.813398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12577.839071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.313495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12579.407929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12579.530598] Lustre: lustre-MDT0000: new disk, initializing [12579.715656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12579.746442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12585.527947] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12598.517545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12598.696292] Lustre: 319676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12598.801433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12598.805605] Lustre: Skipped 1 previous similar message [12598.894611] Lustre: lustre-MDT0001: new disk, initializing [12599.026348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12599.083511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12599.104561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12605.570504] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12616.870638] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12625.315893] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12634.533665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12634.966753] Lustre: lustre-OST0000: new disk, initializing [12634.984712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12634.996280] Lustre: 321647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12635.168630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12643.289679] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12644.464268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12644.489660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12644.755599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12656.669795] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12659.156941] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0 [12660.471125] Lustre: Unmounted lustre-client [12674.543812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.549555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12674.556587] Lustre: Skipped 1 previous similar message [12674.564494] Lustre: Skipped 1 previous similar message [12679.663729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12679.672431] Lustre: Skipped 1 previous similar message [12680.001603] Lustre: server umount lustre-OST0000 complete [12686.306111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12686.322104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12686.339862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12689.893920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12689.904259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12691.591082] Lustre: server umount lustre-MDT0000 complete [12696.398297] LustreError: 319670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598786 with bad export cookie 3867706889681393794 [12696.401520] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12696.416874] LustreError: 319670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12696.952574] Lustre: server umount lustre-MDT0001 complete [12709.783615] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [12712.795472] Key type lgssc unregistered [12718.240667] LNet: 323810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12718.255282] LNetError: 323810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12788.978911] LNet: Removed LNI 192.168.204.108@tcp [12790.018155] Key type .llcrypt unregistered [12790.020434] Key type ._llcrypt unregistered [12812.985877] Key type ._llcrypt registered [12812.988444] Key type .llcrypt registered [12813.177993] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [12831.823176] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [12833.090344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12833.210502] alg: No test for adler32 (adler32-zlib) [12834.496483] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [12834.850528] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [12836.583153] Key type lgssc registered [12838.082951] Lustre: Echo OBD driver; http://www.lustre.org/ [12876.244785] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 05:02:43 (1788598963) [12885.015363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12885.052219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.575044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12886.626127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12886.700796] Lustre: lustre-MDT0000: new disk, initializing [12886.775287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12886.803381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12892.261695] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12905.132758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12905.324989] Lustre: 327042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12905.381133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12905.386770] Lustre: Skipped 1 previous similar message [12905.478709] Lustre: lustre-MDT0001: new disk, initializing [12905.564237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12905.620196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12905.653315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12911.752586] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12923.152696] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12929.818860] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12938.242554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12938.925222] Lustre: lustre-OST0000: new disk, initializing [12938.935061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12938.944629] Lustre: 329012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12939.128066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12946.586815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12946.600084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12946.747397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12948.876103] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12959.176871] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12977.635282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12977.636017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12977.647709] LustreError: Skipped 1 previous similar message [12977.650354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12977.669320] Lustre: Skipped 1 previous similar message [12977.702711] Lustre: Skipped 1 previous similar message [12982.843987] Lustre: server umount lustre-OST0000 complete [12987.365090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12987.374404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12987.398070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12992.996505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12993.007994] Lustre: Skipped 2 previous similar messages [12998.113133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12998.119633] Lustre: Skipped 1 previous similar message [13001.183101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13001.780983] Lustre: server umount lustre-MDT0000 complete [13003.238246] LustreError: 327049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.267225] LustreError: 327049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13006.333238] LustreError: 327034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599096 with bad export cookie 15860042023492959894 [13006.350176] LustreError: 327034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13006.353758] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13006.930827] Lustre: server umount lustre-MDT0001 complete [13019.502978] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13023.169280] Key type lgssc unregistered [13023.496690] LNet: 330920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13023.508760] LNetError: 330920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13023.522913] LNet: Removed LNI 192.168.204.108@tcp [13024.547059] Key type .llcrypt unregistered [13024.549347] Key type ._llcrypt unregistered [13049.723736] Key type ._llcrypt registered [13049.728463] Key type .llcrypt registered [13049.921907] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13051.720883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13051.774956] alg: No test for adler32 (adler32-zlib) [13053.016753] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [13053.348355] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [13055.104202] Key type lgssc registered [13056.657820] Lustre: Echo OBD driver; http://www.lustre.org/ [13066.003528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13066.039647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13067.661147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13073.092225] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13077.993174] LustreError: 332191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13078.011979] LustreError: 332191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13083.119609] LustreError: 332190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13083.167173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13083.627902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13088.407890] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13096.190558] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13103.407304] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13112.256087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13112.643597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13118.775344] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13127.686286] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13151.104767] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13153.149535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13160.416723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13160.433372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13160.468617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13162.977778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13162.981167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13162.998597] Lustre: Skipped 1 previous similar message [13163.013761] Lustre: Skipped 3 previous similar messages [13164.650579] Lustre: server umount lustre-MDT0000 complete [13165.577700] LustreError: 333435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.098778] LustreError: 332190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.128789] LustreError: 332190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13169.047079] LustreError: 332173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599258 with bad export cookie 7655515932695104304 [13169.057987] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13169.080014] LustreError: 332173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13170.716938] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [13173.218466] LustreError: 332185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.225519] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13173.247097] LustreError: 332185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13173.253264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13175.634958] Lustre: server umount lustre-MDT0001 complete [13180.272831] Lustre: server umount lustre-OST0000 complete [13191.013546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13196.975101] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13200.296819] Lustre: 336395:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.108@tcp [13203.150535] Lustre: server umount MGS complete [13212.769025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13213.287030] LustreError: 336967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13213.305543] LustreError: 336967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13213.370278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13219.264391] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13229.642438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13230.094111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13235.264822] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13245.845045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13246.362812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13253.688250] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13262.958295] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13265.256809] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13272.072506] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13274.209954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13280.224448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13280.233274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13280.244669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13280.249619] Lustre: Skipped 1 previous similar message [13281.783130] Lustre: server umount lustre-OST0000 complete [13289.443394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13289.466657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13289.474940] Lustre: Skipped 2 previous similar messages [13289.506313] Lustre: Skipped 2 previous similar messages [13301.217377] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13301.747969] Lustre: server umount lustre-MDT0000 complete [13304.803162] LustreError: 338806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13304.827452] LustreError: 338806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13306.096310] LustreError: 338424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599396 with bad export cookie 7655515932695105144 [13306.102663] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13306.113189] LustreError: 338424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13306.732752] Lustre: server umount lustre-MDT0001 complete [13319.098862] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13322.093665] Key type lgssc unregistered [13322.374380] LNet: 340406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13322.380361] LNetError: 340406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13322.396127] LNet: Removed LNI 192.168.204.108@tcp [13323.359919] Key type .llcrypt unregistered [13323.363566] Key type ._llcrypt unregistered [13343.526746] Key type ._llcrypt registered [13343.528456] Key type .llcrypt registered [13343.662809] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [13359.617070] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13360.757731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13360.855246] alg: No test for adler32 (adler32-zlib) [13362.134373] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [13362.374681] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [13364.159779] Key type lgssc registered [13365.240833] Lustre: Echo OBD driver; http://www.lustre.org/ [13408.854209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13408.864287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13410.328334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13410.405696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13410.579593] Lustre: lustre-MDT0000: new disk, initializing [13410.791945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13410.854527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13417.057716] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13432.456966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13432.709653] Lustre: 343568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13432.828325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13432.836040] Lustre: Skipped 1 previous similar message [13433.029608] Lustre: lustre-MDT0001: new disk, initializing [13433.215702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13433.300506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13433.320919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13438.531913] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13450.029931] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13456.827341] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13463.629266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13463.875334] Lustre: lustre-OST0000: new disk, initializing [13463.880885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13463.891746] Lustre: 345541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13464.082332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13469.896182] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13472.867676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13472.883271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13473.009721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13480.575745] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13486.808787] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13487.186812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13493.368323] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13493.593207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13498.851068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13498.851897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13498.867576] LustreError: Skipped 1 previous similar message [13498.868838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13498.902281] Lustre: Skipped 1 previous similar message [13502.528636] Lustre: server umount lustre-OST0000 complete [13509.089289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13509.090467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13509.101333] Lustre: Skipped 1 previous similar message [13509.108824] Lustre: Skipped 1 previous similar message [13512.798131] Lustre: server umount lustre-MDT0000 complete [13516.919977] LustreError: 343561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599606 with bad export cookie 6841215528964669710 [13516.929058] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13516.935767] LustreError: 343561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13517.257571] Lustre: server umount lustre-MDT0001 complete [13525.411324] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 05:13:33 (1788599613) [13541.067728] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [13552.604831] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13590.656461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13590.981992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13591.024762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13591.124544] Lustre: lustre-MDT0000: new disk, initializing [13591.244301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13591.277164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13596.114597] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13607.600173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13607.740547] Lustre: 350303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13607.786909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13607.791030] Lustre: Skipped 1 previous similar message [13607.855438] Lustre: lustre-MDT0001: new disk, initializing [13607.919634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13607.950186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13607.961364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13613.165317] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13624.492676] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13631.735349] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13640.649513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13641.077471] Lustre: lustre-OST0000: new disk, initializing [13641.082378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13641.090684] Lustre: 352272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13641.210136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13648.960839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13648.979032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13649.146822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13649.305974] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13660.431962] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13705.698209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13705.700137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13705.707151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13705.724952] Lustre: Skipped 1 previous similar message [13705.737432] Lustre: Skipped 1 previous similar message [13709.905507] Lustre: server umount lustre-OST0000 complete [13715.425834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13715.443146] LustreError: Skipped 1 previous similar message [13715.452523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13715.475381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13717.290642] Lustre: server umount lustre-MDT0000 complete [13721.059350] LustreError: 350310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13721.077047] LustreError: 350310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13722.133299] LustreError: 353152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599812 with bad export cookie 6841215528964670592 [13722.143102] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13722.146567] LustreError: 353152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13722.563880] Lustre: server umount lustre-MDT0001 complete [13736.984714] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13740.263800] Key type lgssc unregistered [13740.575079] LNet: 354448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13740.581368] LNetError: 354448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13740.597949] LNet: Removed LNI 192.168.204.108@tcp [13741.501922] Key type .llcrypt unregistered [13741.505321] Key type ._llcrypt unregistered [13860.071567] Key type ._llcrypt registered [13860.075814] Key type .llcrypt registered [13860.294558] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13860.657383] Key type .llcrypt unregistered [13860.661927] Key type ._llcrypt unregistered [13952.112169] Key type ._llcrypt registered [13952.114430] Key type .llcrypt registered [13952.292563] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13952.691493] Key type .llcrypt unregistered [13952.695576] Key type ._llcrypt unregistered [13985.189452] Key type ._llcrypt registered [13985.194028] Key type .llcrypt registered [13985.339522] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13985.834543] Key type .llcrypt unregistered [13985.838316] Key type ._llcrypt unregistered [14019.825376] Key type ._llcrypt registered [14019.829832] Key type .llcrypt registered [14019.989993] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14020.409344] Key type .llcrypt unregistered [14020.412910] Key type ._llcrypt unregistered [14107.504693] Key type ._llcrypt registered [14107.509614] Key type .llcrypt registered [14107.620266] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14107.893379] Key type .llcrypt unregistered [14107.895945] Key type ._llcrypt unregistered [14137.163592] Key type ._llcrypt registered [14137.166822] Key type .llcrypt registered [14137.325145] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14137.693370] Key type .llcrypt unregistered [14137.708212] Key type ._llcrypt unregistered [14168.377540] Key type ._llcrypt registered [14168.379912] Key type .llcrypt registered [14168.578391] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14168.966360] Key type .llcrypt unregistered [14168.971374] Key type ._llcrypt unregistered [14202.245242] Key type ._llcrypt registered [14202.249186] Key type .llcrypt registered [14202.481273] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14202.922297] Key type .llcrypt unregistered [14202.927541] Key type ._llcrypt unregistered [14234.566580] Key type ._llcrypt registered [14234.569627] Key type .llcrypt registered [14234.680431] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14235.002426] Key type .llcrypt unregistered [14235.004166] Key type ._llcrypt unregistered [14269.474786] Key type ._llcrypt registered [14269.477092] Key type .llcrypt registered [14269.598227] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14269.863796] Key type .llcrypt unregistered [14269.865926] Key type ._llcrypt unregistered [14333.970154] Key type ._llcrypt registered [14333.972176] Key type .llcrypt registered [14334.068756] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14334.384292] Key type .llcrypt unregistered [14334.386296] Key type ._llcrypt unregistered [14365.819485] Key type ._llcrypt registered [14365.821993] Key type .llcrypt registered [14365.952569] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14366.255554] Key type .llcrypt unregistered [14366.257587] Key type ._llcrypt unregistered [14403.753592] Key type ._llcrypt registered [14403.757942] Key type .llcrypt registered [14403.912171] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14404.336265] Key type .llcrypt unregistered [14404.341158] Key type ._llcrypt unregistered [14469.814813] Key type ._llcrypt registered [14469.817640] Key type .llcrypt registered [14469.932929] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [14489.996032] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [14491.123495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14491.274974] alg: No test for adler32 (adler32-zlib) [14492.481515] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [14492.837041] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [14494.623172] Key type lgssc registered [14495.863476] Lustre: Echo OBD driver; http://www.lustre.org/ [14540.313276] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14540.348615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14541.644630] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14541.675781] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14541.788356] Lustre: tst250fs-MDT0000: new disk, initializing [14541.955350] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14541.974880] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14546.943304] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14558.720901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14558.901785] Lustre: 374749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14559.030521] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14559.034733] Lustre: Skipped 1 previous similar message [14559.106086] Lustre: tst250fs-MDT0001: new disk, initializing [14559.209385] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14559.246071] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14559.265668] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14564.378527] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14569.881395] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14577.771194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14578.138454] Lustre: tst250fs-OST0000: new disk, initializing [14578.147329] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14578.171129] Lustre: 376383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14578.311868] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14579.707889] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14579.733315] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14579.963888] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14585.793220] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14597.557374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14597.804265] Lustre: tst250fs-OST0001: new disk, initializing [14597.808323] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14597.814854] Lustre: 377251:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14597.906560] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14605.647086] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14607.479584] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14607.496586] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14607.600211] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14611.918650] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14614.524558] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14614.546114] Lustre: Skipped 1 previous similar message [14638.577963] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14638.583966] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14638.600885] Lustre: Skipped 1 previous similar message [14642.997563] Lustre: server umount tst250fs-MDT0000 complete [14647.375994] LustreError: 377867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600737 with bad export cookie 9498416486058172578 [14647.387343] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14647.400674] LustreError: 377867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14648.057951] Lustre: server umount tst250fs-MDT0001 complete [14662.588437] Lustre: server umount tst250fs-OST0000 complete [14664.674825] Lustre: 372973:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788600738/real 1788600738] req@ffff9af16e6dce00 x1875483643942784/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788600754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14664.719382] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14664.734425] Lustre: Skipped 2 previous similar messages [14667.156931] Lustre: server umount tst250fs-OST0001 complete [14689.557616] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [14698.887265] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [14735.663916] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 05:33:42 (1788600822) [14737.805285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14758.167579] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [14771.374715] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [14806.383535] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14541 sec ========= 05:34:53 (1788600893) [14808.378739] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:34:55 (1788600895) === [14812.381595] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:34:59 (1788600899) === [14856.320103] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14860.047718] Key type lgssc unregistered [14860.586782] LNet: 386397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14860.594529] LNetError: 386397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14860.624181] LNet: Removed LNI 192.168.204.108@tcp [14861.639637] Key type .llcrypt unregistered [14861.652944] Key type ._llcrypt unregistered